Commit Graph

20 Commits

Author SHA1 Message Date
Adam Gibson 722592d8a9 Update for publishing to snapshots (#10211)
* fix extra quote inserted by intellij

* Fix path

* fix executable permissions usage

* improve directory resolution and restore some files from the previous PR

* remove old unused dep versions
Update to new march 2024 central release plugin

* update central publishing version

* temp disable mirror

* fix hallucinated version

* update maven invoker version for jetspeed

* remove old build properties
add better directory resolution for copy flatc

* fix helpers sources for cpu (api updates)
perform maven antrun upgrade

* more fixes for onednn helpers

* update more antrun usage

* fix pooling param usage

* remove findbugs on the 2 files it was used on

* debug odd build path issues

* try again

* remove bad ls

* try again

* another attempt

* Add troubleshooting logs again

* Remove specific gcc versions

* update build

* cd to original directory instead

* Remove enforcer check

* remove debug steps

* remove extra steps

* update linux, android and cuda versions

* update arm usage defaults

* remove 32 bit builds

* alternative path

* alternative path get rid of old tools

* get rid of over simplified bootstrap libnd4j download usage

* remove extra cmake

* remove manual path setting

* arm compilation fix

* debug armcompute

* update android-x86_64 openblas

* upgrade gcc

* minor armcompute const changes

* update openblas version

* fix armcompute paths

* remove ls

* update cuda versions and envrionment variables

* fix syntax errors

* fix syntax errors

* fix syntax errors

* fix syntax errors

* another rev

* update msys command

* rev

* fix classifier

* rev android-x86_64

* rev android-x86_64

* update cuda env passing

* fix linux-arm64 syntax error

* cuda rev

* cuda rev

* update arm else if branches for matching

* remove external PS script?

* remove external PS script?

* remove external PS script?

* improve architecture detection

* update list operation conv2d armcompute

* add flatbuffer generated code

* remove old maven auth

* remove old legacny average and accumulate

* remove cache due to 422 error

* ensure armcompute is optional

* ensure armcompute is optional

* disk space clean up on all workflows

* deal with windows service issue

* ensure nvcc is on path

* Add arm64 protoc
Update cuda paths

* convert to use numeric types only mitigating lld errors found by clang

* Add arm64 protoc
Update cuda paths

* fix syntax error

* fix overriding properties causing libnd4j not to be built with cuda

* update windows version

* try to update cl.exe paths

* update debugging for nvcc

* update cuda flags to work with linux/windows

* remove extra cuda install

* fix duplicate sources

* more unsupported compiler changes

* remove unneeded source and javadoc

* better cudnn detection

* update where we put unsupported compiler

* Collapse cmake logic

* Collapse cmake logic

* clean up consolidated file

* flatbuffers fix

* fix hallucinated paths

* generate flatbuffers by default

* fix hallucinated paths

* ensure imports present

* fix flatc target order

* fix elif syntax

* more rearrange

* fix git tag

* update template to allow proper configuration generation

* remove guard

* change quotes

* change quotes

* test default

* fix missing functions

* rearrange dependencies

* set the engine

* reintegrate some old cuda logic

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* Add back in function defs

* update api to use proper cuda versions

* update api to use proper cuda versions

* update api to use proper cuda versions

* Add back in function defs

* delete old paths

* aDd back include_ops.h generation

* delete old paths

* reinroduce old comand

* delete old paths

* delete old paths

* better tar unpacking

* fix for suffixes

* Add missing cpu sources

* fix duplicate profile

* change mac image

* change mac image

* get rid of old gpg key step

* get rid of javadoc on mac

* remove unneeded deps for mac

* update arm compute path

* remove old command checks

* remove gpg key step

* clang specific type erro

* clang specific type erro

* add another missing cpu path

* add another missing cpu path

* fix helpers sources ordering

* fix helpers sources ordering

* Add back createFromDescriptor

* fix helpers sources ordering

* fix helpers sources ordering

* fix helpers sources ordering

* fix helpers sources ordering

* Add back createFromDescriptor

* add debug for onednn

* add debug for onednn

* SFINAE for type aliases

* SFINAE for type aliases

* SFINAE for type aliases

* SFINAE for type aliases

* ensure we tell compiler we're ok with certain apple version minimums

* ensure we tell compiler we're ok with certain apple version minimums

* ensure we tell compiler we're ok with certain apple version minimums

* ensure we tell compiler we're ok with certain apple version minimums

* fix imports

* update linker path

* fix lock type usage

* change order of sources

* share mutex types

* share mutex types

* share mutex types

* update linker path

* share mutex types

* share mutex types

* share mutex types

* decrease type pairs for sort

* decrease type pairs for sort

* decrease type pairs for sort

* decrease type pairs for sort

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* update the special methods to use combinations

* standardize output paths

* fix template paths

* refactor compiler flags

* update onednn to use similar approach to armcompute

* refactor compiler flags

* fix paths

* fix paths

* fix paths

* fix paths

* fix paths

* fix paths

* fix paths

* fix paths

* change target expand types

* change target expand types

* add new pairwise types

* add new pairwise types

* add new pairwise types

* update linker paths

* add new pairwise types

* add new pairwise types

* add new pairwise types

* fix strings with transform

* fix strings with transform

* update linker paths

* update default values for libnd4j.outputPath

* update default values for libnd4j.outputPath

* update default values for libnd4j.outputPath

* update pom.xml namespaces

* Update .github/workflows/build-deploy-linux-cuda-12.6.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .github/workflows/build-deploy-linux-x86_64.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-05-08 04:52:38 +09:00
Adam Gibson 7051233bb4 Clean up pom.xml across modules: logging (#10075)
Remove old test in deeplearning4j-common-tests
Fix shape of created arrays in iris utils
2024-08-02 16:41:58 +09:00
Adam Gibson b57991671d Fix jdk 8 target + failing tests (#9818)
* Fix jdk 8 target

* Fix kotlin vversion in tests

* Fix epsilon op type
Remove unmaintained lint plugin (causes builds to fail on jdk 11 due to java 9 issues)
Disables long running tests

* Test cleanups
Fix up ArrayCacheMemoryMgr: Handles default values with respect to the static context now including the tests.
Delete old misc tests that aren't consistent or need to be rewritten.
Delete old misc commented code
Fix up tests that had failures due to the old 2d vector assumption. These failures were related to  bias shape assertions.
2022-10-17 22:11:57 +09:00
Adam Gibson 01f340ea13 Misc 2d grad fixes, Spark test fixes test refactoring (#9735)
* Fix up create_view tests, assign operation for views

* Misc shape related fixes for gradients and tests related to old 2d assumption with vectors

* Ignore missing op

* Fix updater get with 1d standardization

* Fix up another 1d test case

* Update TestRegressionTest080.java

* Update SlicingTests.java

* Fix up booleanindexing backtracklinesearch test cases

* Fix up test assertion

* Remove old tensor array logic for tensorarray remove, fix test

* Address nits, remove noninplacevalidation listener

* Remove tests with old assumptions, migrate arrowserde test to platform-tests

* Add testresources profile to nd4j-presets-common

* More test collapsing

* Collapse tests

* Remove old profiles

* Delete old aurora backend, misc pom clean up

* Delete old test resources profiles

* Update datavec-api test locations

* More test migrations

* More tests

* Migrate tests for spark

* Collapse spark parameter server tests

* Remove old tests

* Collapse omnihub

* Rename tests for compilation errors

* Graph package moving

* Add missing dep

* nlp test adjustments

* Update names in platform-tests datavec local modules

* Add missing nlp imports

* Remove hard coded test, add deepwalk import

* Update pom.xml

* Update pom.xml

* Update pom.xml

* Add mockito dep

* Update TestGraphLoading.java

* Update pom.xml

* Remove deprecated tests, add missing dep

* Clean up intellij accidental refactoring

* Update AeronUdpTransportTest.java

* Update EndingPreProcessorTest.java

* More test updates

* More test refactoring

* More assertion clean up, remove deprecated test

* More test clean up

* Remove more deprecated tests

* More test updates

* More test updates

* More tests

* Update WeightedWalkerTest.java

* Update spark versions, dependencies

* Handle spark upgrade, add a set closeable false to prevent spark from closing arrays early
2022-07-25 07:07:14 +09:00
Adam Gibson 3feca6d831 Python4j performance optimizations (#9688)
* WIP: adding python interpreter and new abstractions

* Add tests for python update

* fix up cuda build

* WIP: minor clean up, add netty shading

* Fix up test execution, ensure python set main thread is restored

* Add caching option for python (makes objects derived available outside of GIL)

* remove unused test case, extra printing
2022-05-11 16:57:35 +09:00
Adam Gibson 8ce2d9df8f Java support Version updates (#9626)
* Add back CI updates

* Fix android compilation error

* Update TopK.java

* Updates project for java 9+

* Remove old grpc server, rename backend packages, change flatbuffers to temp konduit fork for cuda compilation

* More renames + remove extra generated classes

* Update 0016 - Java 9+ Support.md

* Add back license from flatbuffers generation

* Remove solr modules, update java 9 module comments (fixed some leaving some ignored such as anything spark related)

* Update pom.xml

* Add cuda compute capabilities to reduce build times

* Add default flags for only building certain modules for cuda on windows

* Add back java 8 support

* Update build-deploy-windows-cuda-11.2.yml

* Update build-deploy-windows-cuda-11.4.yml

* Update build-deploy-windows-cuda-11.2.yml

* Update ccs, remove defaults

* Update build-deploy-linux-cuda-11.4.yml

* Add module info

* Update IndexedTail.java

* Add also make and remove deeplearning4j-cuda from build

* Add also make and remove deeplearning4j-cuda from build

* Update IndexedTail.java

* Add missing cuda modules

* Remove old unused concurrency module
2022-02-09 19:25:39 +09:00
Adam Gibson 5826a44294 Update build versions, cuda versions 2021-08-25 07:59:36 +09:00
Adam Gibson 7f6acc1d85 Remove duplicate declarations for cuda single threading 2021-05-04 07:50:48 +09:00
Adam Gibson cc9f62fbd3 Update pom.xmls to add dynamic cuda versions to tests 2021-05-02 21:31:25 +09:00
Adam Gibson 42926dc7a5 Pom changes for junit 5 upgrade 2021-04-01 15:59:15 +09:00
agibsonccc 94b14a9c74 Update maven profiles in python4j allowing tests to run 2021-03-18 12:35:33 +09:00
agibsonccc e0077c38a9 More junit 4 removal, all tests compile. FIxed parameterized test invocation. Deleted nd4j-parameter-server-status that used play 2021-03-17 20:04:53 +09:00
agibsonccc fa1a31c877 Upgrade dl4j to junit 5 2021-03-15 13:02:01 +09:00
agibsonccc 1eaee7f6d9 Copyright updates, removal of extra nlp modules 2021-02-18 11:46:53 +09:00
agibsonccc c715aea405 Update LICENSE 2021-02-01 17:47:29 +09:00
agibsonccc 65c6a9a42e Dev commits 2021-02-01 14:31:20 +09:00
Samuel Audet 029b84e2b7 Development updates (#9053)
* RL4J: Add generic update rule (#502)

Signed-off-by: Alexandre Boulanger <aboulang2002@yahoo.com>

* Shyrma reduce (#481)

* - start working on improving of cpu legacy code for reduce ops

Signed-off-by: Yurii <iuriish@yahoo.com>

* - further work on improving legacy loops

Signed-off-by: Yurii <iuriish@yahoo.com>

* - still working on improving reduce ops

Signed-off-by: Yurii <iuriish@yahoo.com>

* - further work on improving reduce ops

Signed-off-by: Yurii <iuriish@yahoo.com>

* - testing speed run of new reduce op

Signed-off-by: Yurii <iuriish@yahoo.com>

* - working on improvement of default loop for reduce op

Signed-off-by: Yurii <iuriish@yahoo.com>

* - update signatures of stuff which calls reduce ops

Signed-off-by: Yurii <iuriish@yahoo.com>

* - make corrections in cuda reduce kernels

Signed-off-by: Yurii <iuriish@yahoo.com>

* - change loop for default case in broadcast legacy ops

Signed-off-by: Yurii <iuriish@yahoo.com>

* - comment some shape stuff

Signed-off-by: Yurii <iuriish@yahoo.com>

* - comment unnecessary prints in RNGtests

Signed-off-by: Yurii <iuriish@yahoo.com>

* - finish to resolve conflicts after master has been merged

Signed-off-by: Yurii <iuriish@yahoo.com>

* - get rid of some compilation mistakes of cuda stuff

Signed-off-by: Yurii <iuriish@yahoo.com>

* - minor changes

Signed-off-by: Yurii <iuriish@yahoo.com>

* - further search for bug causing crash on java test

Signed-off-by: Yurii <iuriish@yahoo.com>

* - add scalar case in reduce_ ... exec stuff

Signed-off-by: Yurii <iuriish@yahoo.com>

* - minor corrections in NAtiveOps.cu

Signed-off-by: Yurii <iuriish@yahoo.com>

* - add switch to scalar case execReduceXD functions

Signed-off-by: Yurii <iuriish@yahoo.com>

* - add support for vectors old shape in ConstantShapeHelper::createShapeInfoWithNoUnitiesForReduce

Signed-off-by: Yurii <iuriish@yahoo.com>

* - correct cuda mirrorPad

Signed-off-by: Yurii <iuriish@yahoo.com>

* - add support for vectors old shape in cuda createShapeInfoWithNoUnitiesForReduce

Signed-off-by: Yurii <iuriish@yahoo.com>

Co-authored-by: raver119 <raver119@gmail.com>

* Add support for CUDA 11.0 (#492)

* Add support for CUDA 11.0

* libnd4j tweaks for CUDA 11

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* bindings update, again?

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* * Update versions of JavaCPP Presets for FFmpeg, OpenBLAS, and NumPy

* update API to match CUDA 8

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* * Update version of JavaCPP Presets for CPython

* C++ updated for cuDNN 8.0

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* one more test

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* one more test

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* one more test

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* 128-bit alignment for workspaces

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* change seed in 1 test

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* Fix dependecy duplication in python4j-parent pom

* Fix group id for in python4j-numpy

* few tests tweaked

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* Remove macosx-x86_64-gpu from nd4j-tests-tensorflow

* few minor tweaks for IndexReduce

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

* one test removed

Signed-off-by: raver119@gmail.com <raver119@gmail.com>

Co-authored-by: raver119@gmail.com <raver119@gmail.com>
Co-authored-by: Serhii Shepel <9946053+sshepel@users.noreply.github.com>

* RL4J: Add SyncTrainer and AgentLearnerBuilder for a few algorithms (#504)

Signed-off-by: Alexandre Boulanger <aboulang2002@yahoo.com>

Co-authored-by: Alexandre Boulanger <44292157+aboulang2002@users.noreply.github.com>
Co-authored-by: Yurii Shyrma <iuriish@yahoo.com>
Co-authored-by: raver119 <raver119@gmail.com>
Co-authored-by: Serhii Shepel <9946053+sshepel@users.noreply.github.com>
2020-07-26 21:59:27 +09:00
Fariz Rahman e9c13ca9f4 Python4j change namespace (org.eclipse->org.nd4j) (#493)
* org.eclipse->org.nd4j

* Fix parent groupid

Signed-off-by: Alex Black <blacka101@gmail.com>

* memview->bytes

* del test

Co-authored-by: Alex Black <blacka101@gmail.com>
2020-06-25 13:05:08 +10:00
Fariz Rahman 9ca679e080 python4j-numpy (#475)
* 'initial'

* 'impl'

* tests

* <T>

* more tests

* scalar fixes

* lazy setup jobs

* more tests

* multithreading wip

* multithreading fix

* bytebuffer working

* nits

* inplace exec fixes

* attempt linux cpu fix

* rollback

* list fixes

* disable gc

* log

* bump jcpp + fixes

* #8985 GradientSharingTrainingTest ignore for logged issue

Signed-off-by: Alex Black <blacka101@gmail.com>

* memview fixes

* fix?

Co-authored-by: Alex Black <blacka101@gmail.com>
2020-06-16 11:43:10 +10:00
Fariz Rahman bde0a4ec98 Python4j (#422)
* types

* pom fix

* basic exec + tests

* safe exec

* exec fixes + tests

* prim tests

* lists and dicts

* collections tests

* list test

* api

* exec and return all vars

* context manager + fixes

* leak fixes

* jobs tests

* gc basic working

* more gc fixed

* copyright headers

* try-catch-finally

* gc fixes

* validate var name (startswith _collapsed..)

* try block refac

* pythonexecutioner nits

* hashset->set

* call() gc fix

* gc fixes

* type check fix

* types fixes

* refacs

* rem numpyarray

* threadsafety check

* private->public

* threadsafe checks

* pythonGC test

* threading fixes + tests

* threading tests+

* threading test fixes

* make PythonException unchecked

* nits

* docstrings

* path fixes
2020-05-21 11:47:12 +10:00