We will tell you how to transfer data files and how to request CMS dataset transfer.
We explained the gsiftp and xroot protocols in the description of the CMS Tier-3 Computing Environment. You can transfer the files to KISTI Tier-3 at any time.
1. Data Transfer
When you know the exact location of the data file
If you know the exact location of the data file, you can process the data transfer very quickly and easily. You can transfer data to the following commands:
xrdcp root://[source of data file]//[Path of source data file [Dest]
The receive and send destination must be a local directory or an XRootD server.
You do not know the location of the data file
If you don't know the location of the data file, you can find the location of the data and transfer it, or use the CMS Any Data, Anytime, Anywhere (AAA) feature to transfer data regardless of the location of the data file.
Find Data Files
It is most convenient to use the dasgoclient or DAS home page to locate the data file.
Currently, CMS AAA does not offer additional features, such as searching for the fastest sites. Therefore, even if you want to transfer files from other sites in the country, there is a possibility of transferring files from Europe or the United States. If you have more files to send, let's search directly for data and put a prefix to transfer the file.
2. Request to send dataset
(1) Rucio (For Public Dataset)
You must be assigned a space quota for T3_KR_KISTI to transfer data through the current lucio command. If you need a quota, please request by e-mail(mailto:cmst3-support@kisti.re.kr) with the following information.
CERN Computing ID
Requested quota capacity (up to 10TB applicable without additional proof)
Sites requesting quotas (T2_KR_KISTI or T3_KR_KISTI)
Enable the rucio command with the following command: (We also provide rucioenv instead.)
###CMSSW 디렉토리 내에서 cmsenv를 하였을 경우 작동 안될 가능성 있음.
source /cvmfs/cms.cern.ch/cmsset_default.sh
source /cvmfs/cms.cern.ch/rucio/setup.sh
export RUCIO_ACCOUNT=<CERN ID>
voms-proxy-init --voms cms
### or below command
rucioenv
Proceed with the request to send the dataset with the command below.
### If you have enough quota,
rucio add-rule cms:<DATASET> 1 T3_KR_KISTI
## If you do not have enough quota,
rucio add-rule --ask-approval cms:<DATASET> 1 T3_KR_KISTI
You can check your transfer request status using below command.
rucio list-rules --account geonmo
Output Format
[rucio ID as HexCode] [User] cms:[DataSet] [Status/ OK, REPL,STUCK] [Tier Center] [COPIES]
OK : Completed files
REPL : Transferring files
STUCK : File being retried because transfer failed
Dataset created and published by an individual user cannot be sent through rucio directly yet. As a result, we provide scripts that make it easy to transfer each file. Please follow the procedure below to transfer the data.
hep-tools Git 저장소로부터 dataset2filelist.sh와 xrdDownload.py 를 본인의 PATH 경로로 다운로드 받습니다.
dataset2filelist.sh 를 이용하여 다운로드 받으려는 데이터셋들의 개별 파일 목록을 작성합니다.
rucio를 호라용하실 때 사용량 및 쿼터 용량 확인이 필요하신 경우가 있습니다. 아래와 같이 확인하시기 바랍니다.
(1) 쿼터 확인
[geonmo@ui20 ~]$ rucio list-account-limits geonmo
/cvmfs/cms.cern.ch/rucio/x86_64/slc7/py2/current/lib/python2.7/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 2 is no longer
supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
from cryptography.hazmat.backends import default_backend
+-------------+-----------+
| RSE | LIMIT |
|-------------+-----------|
|T2_KR_KISTI|20.000TB||T3_KR_KISTI|20.000TB|+-------------+-----------++------------------+---------+|RSEEXPRESSION|LIMIT||------------------+---------|+------------------+---------+
