This documentation was auto-translated from German by AI and may contain inaccuracies.

Transfer of Large Data Volumes¶

For the transfer of large data volumes, the submission option 'Transfer a large dataset' must be selected. Here, a password-protected share can be created via the TU Dresden service Datashare (Nextcloud), which allows for easier transfer of larger data volumes than via the OPARA website. For members of the TU Dresden, there is also the option to transfer data directly to the internal group drive. Please note in any case the notes in this section regarding the number and size of the files.

Transfer via TUD Datashare (Nextcloud)¶

Via the form component 'Data Transfer via TUD Datashare', a personal password-protected share can be created and released (button 'Generate Storage Access'). After completing the data transfer, the share must be closed ('Close Storage Access').

It should be noted that the maximum file size for upload in Datashare is 5 GB. Larger files must be split accordingly (e.g. via zip -s, split -b or 7Zip (Windows)). Before import, we can possibly reassemble split files. To do this, please specify the method used and how the files can be reassembled in the 'Import Information' comment field.

Further information: Datashare in the ZIH Service Catalog, Datashare FAQ

Transfer via TUD Group Drive¶

This option is exclusively available to members of the TU Dresden: Within the TU Dresden network, data can also be transferred directly to the shared group drive of OPARA.

Please proceed as follows:

Create a local directory, e.g. <date>_<name>
Package your data as zip or tar archives in this directory
Copy this directory completely via the data gateway to the shared directory on the OPARA group drive: dgw.zih.tu-dresden.de:/glw/opara/import/
Then enter the name of the created directory in the form field 'Name of the transferred directory'.

1. Transfer from the TUD Campus Network¶

For a Linux shell, the commands for rsync or scp might look like this:

> rsync -avzh --progress <local directory> <ZIH login>@dgw.zih.tu-dresden.de:/glw/opara/import/
> scp -r <local directory> <ZIH login>@dgw.zih.tu-dresden.de:/glw/opara/import/

For transferring from Windows, additional tools such as PuTTY/pscp are required. On the Windows shell, the command might look like this:

> pscp -sftp -r <local directory> <ZIH login>@dgw.zih.tu-dresden.de:/glw/opara/import/

Alternatively, graphical tools like FileZilla (Windows and Linux), WinSCP (Windows) or others can be used. For access, the following information is required:

Server: sftp://dgw.zih.tu-dresden.de/glw/opara/import/
Username: <zih login>
Password: **********
Port: -free- or 22

2. Transfer from the ZIH High-Performance Computers¶

Data transfers directly from the HPC systems are possible with the dt-tools via the Datamover nodes. For asynchronous data transfer, a separate SSH key without a password must be created. The entire process is as follows:

Creating the SSH key on the HPC cluster:

barnard$ ssh-keygen -t ed25519 -f ~/.ssh/opara -C "data transfer key for $(whoami)@$(hostname)" -P "" # Passphrase must be empty

The new key is copied to the common home directory on login nodes and data gateway nodes:

barnard$ ssh-copy-id -i ~/.ssh/opara <zih login>@login1.zih.tu-dresden.de

The host key of the data gateway must be made known locally:

barnard$ ssh-keyscan -t rsa dgw.zih.tu-dresden.de >> ~/.ssh/known_hosts

When using the dtrsync command, the information about the new SSH key must be provided:

barnard$ dtrsync -av -e \"ssh -i \$HOME/.ssh/opara\" /data/walrus/<your_project> dgw.zih.tu-dresden.de:/glw/opara/import/

The backslashes and quotation marks in the -e parameter must be set exactly.

Alternatively, a dedicated SSH config for the data gateway node can be created, e.g. for frequent use. The procedure would be:

barnard$ cat ~/.ssh/config
  Host dgw.zih.tu-dresden.de
      User <zih login>
      IdentityFile ~/.ssh/opara

With this configuration, the dtrsync command can be executed without additional parameters for the SSH key:

barnard$ dtrsync -av /data/walrus/<your_project> dgw.zih.tu-dresden.de:/glw/opara/import/

Completion of Data Import¶

In the dropdown menu 'Import Information', select whether the transferred data should be archived or published. In the comment field, any notes on further processing of the data can be provided.

After submitting the form via 'Deposit', the data will be imported into OPARA. A new submission will be created, which can then be edited and completed in the web interface like a normal submission. You will receive an email once the import has been completed and the new submission is available for further processing.