Dataverse 6.0

This is a platform upgrade release. Payara, Solr, and Java have been upgraded. No features have been added to the Dataverse software itself. Only a handful of bugs were fixed.

Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project!

Release Highlights (Major Upgrades, Breaking Changes)

This release contains major upgrades to core components. Detailed upgrade instructions can be found below.

Runtime

• The required Java version has been increased from version 11 to 17.
  • See PR #9764 for details.
• Payara application server has been upgraded to version 6.2023.8.
  • This is a required update.
  • Please note that Payara Community 5 has reached end of life
  • See PR #9685 and PR #9795 for details.
• Solr has been upgraded to version 9.3.0.
  • See PR #9787 for details.
• PostgreSQL 13 remains the tested and supported version.
  • That said, the installer and Flyway have been upgraded to support PostgreSQL 14 and 15. See the PostgreSQL section of the Installation Guide and PR #9877 for details.

Development

• Removal of Vagrant and Docker All In One (docker-aio), deprecated in Dataverse v5.14. See PR #9838 and PR #9685 for details.
• All tests have been migrated to use JUnit 5 exclusively from now on. See PR #9796 for details.

Installation

If this is a new installation, please follow our Installation Guide. Please don't be shy about asking for help if you need it!

Once you are in production, we would be delighted to update our map of Dataverse installations around the world to include yours! Please create an issue or email us at [email protected] to join the club!

You are also very welcome to join the Global Dataverse Community Consortium (GDCC).

Upgrade Instructions

Upgrading requires a maintenance window and downtime. Please plan ahead, create backups of your database, etc.

These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 5.14.

Upgrade from Java 11 to Java 17

Java 17 is now required for Dataverse. Solr can run under Java 11 or Java 17 but the latter is recommended. In preparation for the Java upgrade, stop both Dataverse/Payara and Solr.

1. Undeploy Dataverse, if deployed, using the unprivileged service account.

sudo -u dataverse /usr/local/payara5/bin/asadmin list-applications

sudo -u dataverse /usr/local/payara5/bin/asadmin undeploy dataverse-5.14

1. Stop Payara 5.

sudo -u dataverse /usr/local/payara5/bin/asadmin stop-domain

1. Stop Solr 8.

sudo systemctl stop solr.service

1. Install Java 17.

Assuming you are using RHEL or a derivative such as Rocky Linux:

sudo yum install java-17-openjdk

1. Set Java 17 as the default.

Assuming you are using RHEL or a derivative such as Rocky Linux:

sudo alternatives --config java

1. Test that Java 17 is the default.

java -version

Upgrade from Payara 5 to Payara 6

If you are running Payara as a non-root user (and you should be!), remember not to execute the commands below as root. Use sudo to change to that user first. For example, sudo -i -u dataverse if dataverse is your dedicated application user.

1. Download Payara 6.2023.8.

curl -L -O https://nexus.payara.fish/repository/payara-community/fish/payara/distributions/payara/6.2023.8/payara-6.2023.8.zip

1. Unzip it to /usr/local (or your preferred location).

sudo unzip payara-6.2023.8.zip -d /usr/local/

1. Change ownership of the unzipped Payara to your "service" user ("dataverse" by default).

sudo chown -R dataverse /usr/local/payara6

1. Undeploy Dataverse, if deployed, using the unprivileged service account.

sudo -u dataverse /usr/local/payara5/bin/asadmin list-applications

sudo -u dataverse /usr/local/payara5/bin/asadmin undeploy dataverse-5.14

1. Stop Payara 5, if running.

sudo -u dataverse /usr/local/payara5/bin/asadmin stop-domain

1. Copy Dataverse-related lines from Payara 5 to Payara 6 domain.xml.

sudo -u dataverse cp /usr/local/payara6/glassfish/domains/domain1/config/domain.xml /usr/local/payara6/glassfish/domains/domain1/config/domain.xml.orig

sudo egrep 'dataverse|doi' /usr/local/payara5/glassfish/domains/domain1/config/domain.xml > lines.txt

sudo vi /usr/local/payara6/glassfish/domains/domain1/config/domain.xml

The lines will appear in two sections, examples shown below (but your content will vary).

Section 1: system properties (under <server name="server" config-ref="server-config">)

   <system-property name="dataverse.db.user" value="dvnuser"></system-property>
   <system-property name="dataverse.db.host" value="localhost"></system-property>
   <system-property name="dataverse.db.port" value="5432"></system-property>
   <system-property name="dataverse.db.name" value="dvndb"></system-property>
   <system-property name="dataverse.db.password" value="dvnsecret"></system-property>

Note: if you used the Dataverse installer, you won't have a dataverse.db.password property. See "Create password aliases" below.

Section 2: JVM options (under <java-config classpath-suffix="" debug-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9009" system-classpath="">, the one under <config name="server-config">, not under <config name="default-config">)

   <jvm-options>-Ddataverse.files.directory=/usr/local/dvn/data</jvm-options>
   <jvm-options>-Ddataverse.files.file.type=file</jvm-options>
   <jvm-options>-Ddataverse.files.file.label=file</jvm-options>
   <jvm-options>-Ddataverse.files.file.directory=/usr/local/dvn/data</jvm-options>
   <jvm-options>-Ddataverse.rserve.host=localhost</jvm-options>
   <jvm-options>-Ddataverse.rserve.port=6311</jvm-options>
   <jvm-options>-Ddataverse.rserve.user=rserve</jvm-options>
   <jvm-options>-Ddataverse.rserve.password=rserve</jvm-options>
   <jvm-options>-Ddataverse.auth.password-reset-timeout-in-minutes=60</jvm-options>
   <jvm-options>-Ddataverse.timerServer=true</jvm-options>
   <jvm-options>-Ddataverse.fqdn=dev1.dataverse.org</jvm-options>
   <jvm-options>-Ddataverse.siteUrl=https://dev1.dataverse.org</jvm-options>
   <jvm-options>-Ddataverse.files.storage-driver-id=file</jvm-options>
   <jvm-options>-Ddoi.username=testaccount</jvm-options>
   <jvm-options>-Ddoi.password=notmypassword</jvm-options>
   <jvm-options>-Ddoi.baseurlstring=https://mds.test.datacite.org/</jvm-options>
   <jvm-options>-Ddoi.dataciterestapiurlstring=https://api.test.datacite.org</jvm-options>

1. Check the Xmx setting in domain.xml.

Under /usr/local/payara6/glassfish/domains/domain1/config/domain.xml, check the Xmx setting under <config name="server-config">, where you put the JVM options, not the one under <config name="default-config">. Note that there are two such settings, and you want to adjust the one in the stanza with Dataverse options. This sets the JVM heap size; a good rule of thumb is half of your system's total RAM. You may specify the value in MB (8192m) or GB (8g).

1. Copy jhove.conf and jhoveConfig.xsd from Payara 5, edit and change payara5 to payara6.

sudo cp /usr/local/payara5/glassfish/domains/domain1/config/jhove* /usr/local/payara6/glassfish/domains/domain1/config/

sudo chown dataverse /usr/local/payara6/glassfish/domains/domain1/config/jhove*

sudo -u dataverse vi /usr/local/payara6/glassfish/domains/domain1/config/jhove.conf

1. Copy logos from Payara 5 to Payara 6.

These logos are for collections (dataverses).

sudo -u dataverse cp -r /usr/local/payara5/glassfish/domains/domain1/docroot/logos /usr/local/payara6/glassfish/domains/domain1/docroot

1. If you are using Make Data Count (MDC), edit :MDCLogPath.

Your :MDCLogPath database setting might be pointing to a Payara 5 directory such as /usr/local/payara5/glassfish/domains/domain1/logs. If so, edit this to be Payara 6. You'll probably want to copy your logs over as well.

1. Update systemd unit file (or other init system) from /usr/local/payara5 to /usr/local/payara6, if applicable.

2. Start Payara.

sudo -u dataverse /usr/local/payara6/bin/asadmin start-domain

1. Create a Java mail resource, replacing "localhost" for mailhost with your mail relay server, and replacing "localhost" for fromaddress with the FQDN of your Dataverse server.

sudo -u dataverse /usr/local/payara6/bin/asadmin create-javamail-resource --mailhost "localhost" --mailuser "dataversenotify" --fromaddress "do-not-reply@localhost" mail/notifyMailSession

1. Create password aliases for your database, rserve and datacite jvm-options, if you're using them.

echo "AS_ADMIN_ALIASPASSWORD=yourDBpassword" > /tmp/dataverse.db.password.txt

sudo -u dataverse /usr/local/payara6/bin/asadmin create-password-alias --passwordfile /tmp/dataverse.db.password.txt

When you are prompted "Enter the value for the aliasname operand", enter dataverse.db.password

You should see "Command create-password-alias executed successfully."

You'll want to perform similar commands for rserve_password_alias and doi_password_alias if you're using Rserve and/or DataCite.

1. Enable workaround for FISH-7722.

The following workaround is for https://github.com/payara/Payara/issues/6337

sudo -u dataverse /usr/local/payara6/bin/asadmin create-jvm-options --add-opens=java.base/java.io=ALL-UNNAMED

1. Create the network listener on port 8009.

sudo -u dataverse /usr/local/payara6/bin/asadmin create-network-listener --protocol http-listener-1 --listenerport 8009 --jkenabled true jk-connector

1. Deploy the Dataverse 6.0 war file.

sudo -u dataverse /usr/local/payara6/bin/asadmin deploy /path/to/dataverse-6.0.war

1. Check that you get a version number from Dataverse.

This is just a sanity check that Dataverse has been deployed properly.

curl http://localhost:8080/api/info/version

1. Perform one final Payara restart to ensure that timers are initialized properly.

sudo -u dataverse /usr/local/payara6/bin/asadmin stop-domain

sudo -u dataverse /usr/local/payara6/bin/asadmin start-domain

Upgrade from Solr 8 to 9

Solr has been upgraded to Solr 9. You must install Solr fresh and reindex. You cannot use your old schema.xml because the format has changed.

The instructions below are copied from https://guides.dataverse.org/en/6.0/installation/prerequisites.html#installing-solr and tweaked a bit for an upgrade scenario.

We assume that you already have a user called "solr" (from the instructions above), added during your initial installation of Solr. We also assume that you have already stopped Solr 8 as explained in the instructions above about upgrading Java.

1. Become the "solr" user and then download and configure Solr.

su - solr

cd /usr/local/solr

wget https://archive.apache.org/dist/solr/solr/9.3.0/solr-9.3.0.tgz

tar xvzf solr-9.3.0.tgz

cd solr-9.3.0

cp -r server/solr/configsets/_default server/solr/collection1

1. Unzip "dvinstall.zip" from this release. Unzip it into /tmp. Then copy the following files into place.

cp /tmp/dvinstall/schema*.xml /usr/local/solr/solr-9.3.0/server/solr/collection1/conf

cp /tmp/dvinstall/solrconfig.xml /usr/local/solr/solr-9.3.0/server/solr/collection1/conf

1. A Dataverse installation requires a change to the jetty.xml file that ships with Solr.

Edit /usr/local/solr/solr-9.3.0/server/etc/jetty.xml, increasing requestHeaderSize from 8192 to 102400

1. Tell Solr to create the core "collection1" on startup.

echo "name=collection1" > /usr/local/solr/solr-9.3.0/server/solr/collection1/core.properties

1. Update your init script.

Your init script may be located at /etc/systemd/system/solr.service, for example. Update the path to Solr to be /usr/local/solr/solr-9.3.0.

1. Start Solr using your init script and check collection1.

The collection1 check below should print out fields Dataverse uses like "dsDescription".

systemctl start solr.service

curl http://localhost:8983/solr/collection1/schema/fields

1. Reindex Solr.

For details, see https://guides.dataverse.org/en/6.0/admin/solr-search-index.html but here is the reindex command:

curl http://localhost:8080/api/admin/index

1. If you have custom metadata blocks installed, you must update your Solr schema.xml to include your custom fields.

For details, please see https://guides.dataverse.org/en/6.0/admin/metadatacustomization.html#updating-the-solr-schema

At a high level you will be copying custom fields from the output of http://localhost:8080/api/admin/index/solr/schema or using a script to automate this.

Potential Archiver Incompatibilities with Payara 6

The Google Cloud and DuraCloud archivers may not work in Dataverse 6.0.

This is due to the archivers' dependence on libraries that include classes in javax.* packages that are no longer available. If these classes are actually used when the archivers run, the archivers would fail. As these two archivers require additional setup, they have not been tested in 6.0. Community members using these archivers or considering their use are encouraged to test them with 6.0 and report any errors and/or provide fixes for them that can be included in future releases.

Bug Fix for Dataset Templates with Custom Terms of Use

A bug was fixed for the following scenario:

• Create a template with custom terms.
• Set that template as the default.
• Try to create a dataset.
• A 500 error appears before the form to create dataset is even shown.

For more details, see issue #9825 and PR #9892

Complete List of Changes

For the complete list of code changes in this release, see the 6.0 Milestone in GitHub.

Getting Help

For help with upgrading, installing, or general questions please post to the Dataverse Community Google Group or email [email protected].