Release 3.0.0
Release notes - Hopsworks - Version 3.0.0
Bug
HOPSWORKS-3255 elastic search - highlight fragment bug
HOPSWORKS-3253 Feature Group Type Enum breaks with Stream Feature Group after upgrade
HOPSWORKS-3252 Disable Kafka Topic deletion for feature group ingestion topics
HOPSWORKS-3246 New kafka topic names not included in the response after updating a deployment
HOPSWORKS-3236 Improve model inference error responses
HOPSWORKS-3235 Use Hudi snapshot query when users doesn't specify as_of in the query param
HOPSWORKS-3233 Timestamp incompatibility Spark/Hive/Hudi
HOPSWORKS-3232 Deleting a model used in existing deployments should not be possible
HOPSWORKS-3228 Bump scikit-learn to v1.1.1
HOPSWORKS-3227 Return valid inference endpoints only
HOPSWORKS-3221 Spark dist files do not respect HopsFS replication factor
HOPSWORKS-3220 HopsKafkaAuthorizer retry and refresh ACLs cache if cannot find any ACLs
HOPSWORKS-3212 Remove pickle file validation in storage-initializer
HOPSWORKS-3209 expectation_suite update is failing
HOPSWORKS-3207 Unexported service log viewer Chef attributes
HOPSWORKS-3204 Feature View can't get query when one of the feature groups was deleted
HOPSWORKS-3197 Feature Store quota enforcement fail with NPE for non-cached Feature Groups
HOPSWORKS-3193 Creating Feature Group from UI without Expectation Suite fails
HOPSWORKS-3191 OAuth not removing group if mapping is empty
HOPSWORKS-3187 Inference with Istio using HSML not enabled in transformers
HOPSWORKS-3185 Api key scopes do not cascade on delete
HOPSWORKS-3183 Online feature vectors are not sorted according to the user input
HOPSWORKS-3171 Rename PYTHON scope to PYTHON_LIBRARIES
HOPSWORKS-3149 Consul should only register the active resource manager
HOPSWORKS-3142 Transformation function does not handle null value
HOPSWORKS-3141 Query filter does not handle DATE type properly
HOPSWORKS-3133 Hopsify log file is in wrong directory
HOPSWORKS-3129 Consul does not detected a failed MySQLd
HOPSWORKS-3126 Should flush new Serving entity to get id
HOPSWORKS-3124 ruby test - always wait for started executions to stop
HOPSWORKS-3121 [FeatureView] Bug report
HOPSWORKS-3111 Hopsworks K8s certificate expiration should not be hardcoded
HOPSWORKS-3097 Make OAuth provider display name should not be mandatory
HOPSWORKS-3093 FileSaver.js package was outdated from Teleborder repository
HOPSWORKS-3090 CVE-2022-26612 - flink client vulnerability
HOPSWORKS-3087 Feature group data preview returns 500 when previewing streaming feature groups
HOPSWORKS-3070 Install custom PyTorch and Torchvision in base python environment
HOPSWORKS-3059 Latest version for monitored library should be null if not available
HOPSWORKS-3058 Hopsfs-mount chown's root directory ownership during upgrade
HOPSWORKS-3048 Python library installation using APIKey fails to get the user of the request
HOPSWORKS-3038 Add lock management in Python search singletons
HOPSWORKS-3031 Uglify failing hopsworks-web build
HOPSWORKS-3019 Filebeats data directory is wrong
HOPSWORKS-3015 flyway validation_rule table broken for 2.3 -> 2.4 -> 2.5 upgrade
HOPSWORKS-3013 Storage Connector Update not working
HOPSWORKS-2994 ModuleNotFoundError when saving code
HOPSWORKS-2993 Update git provider secrets throws an secret already exists exception
HOPSWORKS-2958 Hopsify should use internal https port to register with Hopsworks
HOPSWORKS-2957 Call hops::default from hopsworks::default
HOPSWORKS-2953 init_prepared_statement throws NoneType exception
HOPSWORKS-2952 update hsfs examples with train_split property
HOPSWORKS-2564 Alert manager config backup timer fails
HOPSWORKS-2100 Cannot enable model serving when Models folder already exists
HOPSWORKS-1932 feature store sharing ignores online feature store
Improvement
HOPSWORKS-3251 Add numFeatureViews to FeaturestoreDTO for front-end
HOPSWORKS-3248 Rename On Demand Feature groups to external feature groups in hsfs
HOPSWORKS-3242 Improve handling of different data types in feature groups across online/offline
HOPSWORKS-3240 Stream Feature Group online enable exception only when trying to change the setting
HOPSWORKS-3231 Support custom external IP for istio ingressgateway
HOPSWORKS-3226 Hops Kafka Authorizer improve synchonization
HOPSWORKS-3225 Throw a warning instead of error while creating empty data featuregroup
HOPSWORKS-3222 Improve Logstash CPU utilization
HOPSWORKS-3217 Set default values for Inference batching and add access to Hopsworks var for Job audience
HOPSWORKS-3216 hops-hadoop-chef hops/nn/heap_size should set only the NN heap
HOPSWORKS-3211 Add endpoint for deployment logs
HOPSWORKS-3206 bump hopshadoop version 3.2.0.7 RC0
HOPSWORKS-3195 Refactor inference endpoints details
HOPSWORKS-3190 Upgrade OpenSearch to 1.3.3
HOPSWORKS-3186 [FeatureView] Feature view training dataset API redesign
HOPSWORKS-3184 feature view search
HOPSWORKS-3181 OAuth update user profile not working
HOPSWORKS-3176 Change the format of the Hopsworks audit logs to JSON.
HOPSWORKS-3173 Kafka store number of partitions and replicas in the database
HOPSWORKS-3169 [hops-hadoop-chef] Expose attribute to configure user limits in Yarn Capacity Scheduler
HOPSWORKS-3161 Hive connection should return better error message when accessing an unshared feature group
HOPSWORKS-3158 Print Job and Feature Store URLs within Hopsworks with correct Hostname
HOPSWORKS-3154 Make nodemanager-graceful-decommission-timeout-secs configurable
HOPSWORKS-3147 Make Kafka clusterinfo endpoint return also the external kafka broker endpoints
HOPSWORKS-3146 Fix delete user end-point
HOPSWORKS-3136 Migrate IAM role chaining docs
HOPSWORKS-3131 there can be only one fg.insert() or fg.save()
HOPSWORKS-3128 Iam role mapping fixes for new UI
HOPSWORKS-3125 [FeatureView] Merge feature view dev branch to master
HOPSWORKS-3123 [FeatureView] remove label from feature view table
HOPSWORKS-3119 Register OAuth user as admin
HOPSWORKS-3118 Create new role for hopsworks as a service user
HOPSWORKS-3114 Enable HTTP 1.0 in istio ingress gateway
HOPSWORKS-3113 Enable Hive session for all Jobs
HOPSWORKS-3112 Documentation for managing services
HOPSWORKS-3109 Allow creation of AWS storage connector without IAM role passthrouh
HOPSWORKS-3104 Fix error messages for services ui
HOPSWORKS-3103 Connect to Hive externally
HOPSWORKS-3102 Support custom domain name in the istio ingress gateway
HOPSWORKS-3100 [FeatureView] Migrate training dataset to feature view
HOPSWORKS-3099 [FeatureView] Java client
HOPSWORKS-3098 [FeatureView] Python client
HOPSWORKS-3096 Multiple expectation suite + remove great expectation matching
HOPSWORKS-3091 Allow users to configure clusterj connection pool/max transactions in onlinefs
HOPSWORKS-3086 change hopsfs small files default to disabled
HOPSWORKS-3084 Add Documentation for Okta group claim
HOPSWORKS-3081 Make OAuth redirect uri provider specific
HOPSWORKS-3079 Support fixed node port for istio-ingressgateway service
HOPSWORKS-3078 Refactor tags sub-resource
HOPSWORKS-3071 allow utf-8 chars for user names/surnames
HOPSWORKS-3069 Allow StreamFeatureGroup in backend to be available offline only
HOPSWORKS-3056 Add hopsworks to monitored libraries
HOPSWORKS-3055 Add chef attribute to limit the size of datanode storage
HOPSWORKS-3052 Add a flag to skip namespace creation when creating a project
HOPSWORKS-3051 Add new Python extra to HSFS for python engine to replace hive
HOPSWORKS-3047 Add chef attribute to control resource manager HTTPs ports
HOPSWORKS-3046 Enable TLS by default
HOPSWORKS-3042 Kerberos fixes for new UI
HOPSWORKS-3041 Bump up RonDB version to 21.04.4
HOPSWORKS-3040 Use informers in the model-serving-authenticator server
HOPSWORKS-3036 Kerberos and LDAP Docs
HOPSWORKS-3034 Move azure OAuth2 doc from cloud to admin
HOPSWORKS-3030 Add hopsworks debug flag
HOPSWORKS-3028 Extend JWT token lifetime
HOPSWORKS-3016 Improve Logstash Resource Utilization Configurability
HOPSWORKS-3014 [consul-chef] Attribute to change Consul client bind interface
HOPSWORKS-3012 Add dummy file in HopsFS for External Training Datasets to support Keywords/Tags
HOPSWORKS-3011 Decrease logstash pipeline workers for each pipeline
HOPSWORKS-3008 Upgrade Fink to 1.14.3 release
HOPSWORKS-3007 Limit TLS versions used by elastic
HOPSWORKS-3006 Add Expectation Suite Logic between common and persistence layer
HOPSWORKS-3005 Duplicate Expectation persistence logic to ExpectationSuite objects
HOPSWORKS-3004 Duplicate Expectation API logic to ExpectationSuite objects
HOPSWORKS-2999 HSFS Hive engine should throw proper errors for all unsupported methods
HOPSWORKS-2998 Hopsfs-mount fails in git container on VM with TLS disabled
HOPSWORKS-2996 Add git integration documentation
HOPSWORKS-2995 Add job configuration documentation
HOPSWORKS-2992 Jupyter update endpoint should update the entire settings
HOPSWORKS-2985 Add ssl_verify flag in condarc file
HOPSWORKS-2974 HSML support in non-kubernetes installations
HOPSWORKS-2970 Fix dependency-check
HOPSWORKS-2966 LDAP and Kerberos fixes for new UI
HOPSWORKS-2960 Migrate User management docs
HOPSWORKS-2956 Move/clean up hive certificates when using HSFS external client
HOPSWORKS-2955 default train_split property to "train"
HOPSWORKS-2949 Add Client IP and User Agent to the Hopsworks audit logs
HOPSWORKS-2921 Migrate Alert docs
HOPSWORKS-2918 Register built-in transformations by defaut for every project in backend
HOPSWORKS-2897 Remove old feature store UI
HOPSWORKS-2832 Support serving custom predictor scripts without model files
HOPSWORKS-2715 Support for 'requested' and 'limited' pod resources configuration
HOPSWORKS-2543 add support to insert_stream() to ingest data into offline fs
HOPSWORKS-2496 Split up JDBC Storage Connector arguments from string into class attributes
HOPSWORKS-2032 Remove unused parameters for small files from hops-hadoop-chef
New Feature
HOPSWORKS-3215 Make Proxy use JWT
HOPSWORKS-3214 [hops-hadoop-chef ]fix wrong param name in hdfs-site.xml
HOPSWORKS-3203 Add training dataset support for GCS connector
HOPSWORKS-3199 Add loging page overwrite option
HOPSWORKS-3120 Quotas enforcement
HOPSWORKS-3054 Stream Feature Group Python Engine ingestion
HOPSWORKS-3010 Add Google BigQuery Connector
HOPSWORKS-2988 REST API Great Expectations DTO
HOPSWORKS-2987 Add create GE store method to ValidationPipeline class
HOPSWORKS-2986 Data Validation Hopsworks3
HOPSWORKS-2947 [FeatureView] Implement PrepareStatementResource
HOPSWORKS-2946 [FeatureView] Implement TransformationResource
HOPSWORKS-2945 [FeatureView] Implement activity endpoints
HOPSWORKS-2944 [FeatureView] Implement keyword related endpoints
HOPSWORKS-2943 [FeatureView] Implement QueryResource
HOPSWORKS-2942 [FeatureView] Implement TagResource
HOPSWORKS-2941 [FeatureView] Implement TrainingDatasetResource
HOPSWORKS-2844 Add autoscaling information to the /metrics endpoint
HOPSWORKS-2741 Add Kafka storage connector
Sub-task
HOPSWORKS-1851 Show feature store info in search
Task
HOPSWORKS-3256 Remove old feature store tour job files
HOPSWORKS-3218 Support custom KServe TensorFlow version
HOPSWORKS-3179 Add catboost and xgboost libraries to base environment
HOPSWORKS-3178 Remove Deequ based data validation in favour of GE
HOPSWORKS-3177 Bump confluent-kafka to 1.8.2
HOPSWORKS-3175 [FeatureView] Add expansion on keywords and tags when getting td by version
HOPSWORKS-3174 Bump-up RonDB version to 21.04.6 and Bump Hops version in Chef to use 3.2.0.7-SNAPSHOT
HOPSWORKS-3153 Rebuild Spark 3.1 after hops dependency change
HOPSWORKS-3145 Chef enabled online feature store by default
HOPSWORKS-3085 Suppress CVE-2021-28165 from zookeeper dependency
HOPSWORKS-3083 Add hopsworks python SDK examples
HOPSWORKS-3076 [hops-hadoop-chef] Bump up Hops ndb version
HOPSWORKS-3018 Set executions_per_job_limit variable in tests
HOPSWORKS-2984 Add expansions to Git API
HOPSWORKS-2976 NullPointerException when logging requests containing queryDTOs
HOPSWORKS-2975 Remove feature store integration test spec
HOPSWORKS-2969 Add [rsa] profile to MySQL dependency in hsfs
HOPSWORKS-2881 Clean old Git support
HOPSWORKS-2879 Upgrade hudi to 0.10.0 release
HOPSWORKS-2673 Upgrade TensorFlow to 2.9 and PyTorch to 1.12