跳转至

TOOLs 高通量测序数据下载之aspera和SRA Toolkit

高通量数据库的数据，下载方法有三种： 1. 常规下载(wget,迅雷、IDM) 2. aspera 3. SRA Toolkit 的prefetch wget下载数据速度很慢，比较容易断。 wget -c 下载地址 保持断点下载 ENA数据下载方法汇总 https://ena-docs.readthedocs.io/en/latest/retrieval/file-download.html#using-wget

下载数据库

优先选择 快速下载fq格式的数据，EBI数据库下载。 从EBI数据库直接获取到aspera的下载代码，复制到本地服务器，可以直接运行。

cd ~/wes_cancer/project/1.raw_fq
ascp -QT -l 300m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR318/008/SRR3182418/SRR3182418_2.fastq.gz . 
ascp -QT -l 300m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR318/003/SRR3182423/SRR3182423_1.fastq.gz .

注意：上述是apera的下载格式，最后面的.代表保存的路径是本目录，一定不能省略，不然会报错。 网络有时候会提示SSH或者UDP错误，可能是服务器的33001端口没有开，打开这个端口即可。

上面这种下载时候，搜索框需要输入SRA号，最近有一个项目的SRA号达到了1785个显然不能收到一个个搜索地址。

新的工具ffq可以用来批量获取下载地址。 ffq github 直接pip安装即可 pip install ffq ffq 参数说明 -o 输出文件名称 -t 指定输入的编号类型，默认是：SRR,可用选项：SRR, ERR, DRR, SRP, ERP, DRP, GSE, DOI ffq输出的结果格式是json

#获取一个SRR的下载地址
ffq  SRR12455819

输出信息如下：直接包括文章的标题摘要，数据类型，数据下载地址和md5

{
    "SRR12455819": {
        "accession": "SRR12455819",
        "experiment": {
            "accession": "SRX8950230",
            "title": "Illumina HiSeq 4000 sequencing; GH0202",
            "platform": "ILLUMINA",
            "instrument": "Illumina HiSeq 4000"
        },
        "study": {
            "accession": "SRP275570",
            "title": "Genebank resequencing of tetraploid cottons",
            "abstract": "Modern cultivated tetraploid cottons contain two species, Gossypium barbadense and Gossypium hirsutum. Among them, G. hirsutum is the significant species cultivated worldwide and contributes more than 90% natural fiber for the industry. To completely reveal the genetic diversity and population divergence within cultivated tetraploid cotton, we resequenced more than 1,700 accessions, which mostly included the South-China landraces, the elite introgression lines, and the obsoleted historic varieties. After integrating them with the major public resequencing data (PRJNA257154, PRJNA336461, PRJNA375965, PRJNA399050, and PRJNA414461), we obtained a whole Genebank variation map contained 3,248 tetraploid cottons (included 2,922 G. hirsutum). This variation map covered more than 1/3 of current G. hirsutum Genebank in China (~9,000 accessions), which could represent the genetic diversity of G. hirsutum."
        },
        "sample": {
            "accession": "SRS7128615",
            "title": "Gossypium hirsutum",
            "organism": "Gossypium hirsutum",
            "attributes": {
                "isolate": "not applicable",
                "cultivar": "GH0202",
                "ecotype": "not applicable",
                "age": "not applicable",
                "dev_stage": "not applicable",
                "geo_loc_name": "not applicable",
                "tissue": "Leaf",
                "BioSampleModel": "Plant",
                "ENA-SPOT-COUNT": "231824444",
                "ENA-BASE-COUNT": "34773666600",
                "ENA-FIRST-PUBLIC": "2020-12-09",
                "ENA-LAST-UPDATE": "2020-12-09"
            }
        },
        "title": "Illumina HiSeq 4000 sequencing; GH0202",
        "files": [
            {
                "url": "ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz",
                "md5": "056333e1c9fba8b9ff930496073cbb95",
                "size": "7552277314"
            }
        ]
    }
}

获取的地址ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz和上面使用aspera下载的地址era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz只有前面的ftp://ftp修改为era-fasp@fasp即可。

批量获取下载地址输出到1个json文件

ffq -o SRA_Acc.json SRR12455818 SRR12455819 SRR12455820 SRR12455821 SRR12455822 SRR12455823 SRR12455824 所有的结果输出到SRA_Acc.json文件

批量获取下载地址，并自动生成ascp的下载命令行

#输出文件在NCBI.json
ffq -o NCBI.json SRR12455818 SRR12455819 SRR12455820 SRR12455821 SRR12455822 SRR12455823 SRR12455824

json2tab.py脚本内容如下： 此脚本目前适配的是SRR和ERP的输出，其他的输出如果不一致，需要自己手动修改。

#!/usr/bin/python3
##用法：python3 json2tab.py NCBI.json
#输入文件是ffq输出的json文件
#输出是"projectID","sampleID","fq_url","size","md5","organism"
import sys
import pandas as pd
import json
#jsonfile = "NCBI.json"
jsonfile = sys.argv[1]
#filename = jsonfile.split('.')[0]
#解析输入的json文件
with open(jsonfile,"r") as load_f:
    load_dict = json.load(load_f)
#输出文件头部信息
#print("projectID","sampleID","fq1_url","fq2_url","size_1","size_1","md5_1","md5_2")
for i in load_dict.keys(): #i是项目的编号，当有多个NCBI的号的时候
    idlist=[]
    for j in load_dict[i].keys():
        idlist.append(j) #把每一个的keys输出到idlist,如果keys里有files选项，则直接输出，如果有runs，需要多解析一层
    if "files" in idlist:
        data_ID=load_dict[i] #此时有files直接解析即可
        for list_url in data_ID['files']:
            print(load_dict[i]['accession'],data_ID['accession'],list_url['url'],list_url['size'],list_url['md5'],data_ID['sample']['organism'],sep='\t')
    elif "runs" in idlist: #此时是需要多解析一层
        for id in load_dict[i]['runs'].keys(): #id是每个项目里，对应的文件的号
            data_ID=load_dict[i]['runs'][id]
            for list_url in data_ID['files']:
                print(load_dict[i]['accession'],data_ID['accession'],list_url['url'],list_url['size'],list_url['md5'],data_ID['sample']['organism'],sep='\t')
    else: #此时可能是其他情况，自行解决
        print("不符合已知的字段，需要自行解析如下内容：")
        print(load_dict[i])

脚本输出的格式如下：

SRR12455818     SRR12455818     ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/018/SRR12455818/SRR12455818.fastq.gz  5425719230      67a1ba5cafd70f6137916589bf9bb437        Gossypium hirsutum
SRR12455819     SRR12455819     ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz  7552277314      056333e1c9fba8b9ff930496073cbb95        Gossypium hirsutum

直接生成转换后的下载地址
python3 json2tab.py NCBI.json >info.list
cat info.list|awk '{print $3}'|sed 's/ftp:\/\/ftp.sra.ebi.ac.uk/ascp -QT -l 30m -P33001 -i \$HOME\/.aspera\/connect\/etc\/asperaweb_id_dsa.openssh era-fasp\@fasp.sra.ebi.ac.uk:/g;s/$/ ./g' >download.sh
nohup bash download.sh & #直接即可下载

download.sh的内容如下：

ascp -QT -l 30m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR124/018/SRR12455818/SRR12455818.fastq.gz .
ascp -QT -l 30m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR124/019/SRR12455819/SRR12455819.fastq.gz .

md5检测下载的数据是否完整

awk '{print $3,$5}' info.list|rev|cut -d "/" -f1|rev|awk '{print $2,$1}' >md5.txt
md5sum -c md5.txt

如果全部ok,就说明数据没问题。

下面内容可忽略

批量获取下载地址分别输出到json文件

ffq -o srr_split --split SRR12455818 SRR12455819 SRR12455820 SRR12455821 SRR12455822 SRR12455823 SRR12455824 所有的结果输出到srr_split的目录里，分别是SRR12455818.json …… SRR12455824.json

不推荐 下载原始的SRA格式，NCBI数据库 在NCBI数据库会获取到Accession List里面是SRR的列表。

使用prefetch下载

##单行手动下载
prefetch SRR3182423

##批量自动化下载
cat SRR_Acc_List.txt | while read id
do
    prefetch ${id} -O  ./
done
数据格式

SRA数据格式，是NCBI数据库的格式，我们下载之后，需要自己手动转换成fq格式。 SRA的数据是每一个SRR数据，是一个文件夹。

回到页面顶部