Authors: Johannes Rainer
Last modified: 2017-09-28 08:47:24
Compiled: Thu Oct 3 09:27:10 2019

1 Fetch EnsDb databases from AnnotationHub

The AHEnsDbs package provides the metadata for all EnsDb SQLite databases in AnnotationHub. First we load/update the AnnotationHub resource.

library(AnnotationHub)
ah <- AnnotationHub()

Next we list all EnsDb entries from AnnotationHub.

query(ah, "EnsDb")
## Using temporary cache /tmp/RtmpdVQx0h/BiocFileCache
## AnnotationHub with 1144 records
## # snapshotDate(): 2019-09-17 
## # $dataprovider: Ensembl
## # $species: Homo sapiens, Ailuropoda melanoleuca, Anolis carolinensis, As...
## # $rdataclass: EnsDb
## # additional mcols(): taxonomyid, genome, description,
## #   coordinate_1_based, maintainer, rdatadateadded, preparerclass,
## #   tags, rdatapath, sourceurl, sourcetype 
## # retrieve records with, e.g., 'object[["AH53185"]]' 
## 
##             title                                      
##   AH53185 | Ensembl 87 EnsDb for Anolis Carolinensis   
##   AH53186 | Ensembl 87 EnsDb for Ailuropoda Melanoleuca
##   AH53187 | Ensembl 87 EnsDb for Astyanax Mexicanus    
##   AH53188 | Ensembl 87 EnsDb for Anas Platyrhynchos    
##   AH53189 | Ensembl 87 EnsDb for Bos Taurus            
##   ...       ...                                        
##   AH73982 | Ensembl 97 EnsDb for Xiphophorus couchianus
##   AH73983 | Ensembl 97 EnsDb for Xiphophorus maculatus 
##   AH73984 | Ensembl 97 EnsDb for Xenopus tropicalis    
##   AH73985 | Ensembl 97 EnsDb for Zonotrichia albicollis
##   AH73986 | Ensembl 79 EnsDb for Homo sapiens

We fetch the EnsDb for species Ciona Intestinalis and ensembl release 87.

qr <- query(ah, c("EnsDb", "intestinalis", "87"))
## Using temporary cache /tmp/RtmpdVQx0h/BiocFileCache
edb <- qr[[1]]
## Using temporary cache /tmp/RtmpdVQx0h/BiocFileCache
## Using temporary cache /tmp/RtmpdVQx0h/BiocFileCache
## downloading 1 resources
## retrieving 1 resource
## Using temporary cache /tmp/RtmpdVQx0h/BiocFileCache
## loading from cache
## require("ensembldb")
## Using temporary cache /tmp/RtmpdVQx0h/BiocFileCache
## Using temporary cache /tmp/RtmpdVQx0h/BiocFileCache
## Using temporary cache /tmp/RtmpdVQx0h/BiocFileCache

To get the genes defined for this species we can simply call.

genes(edb)
## GRanges object with 17153 ranges and 7 metadata columns:
##                      seqnames      ranges strand |            gene_id
##                         <Rle>   <IRanges>  <Rle> |        <character>
##   ENSCING00000017842        1   1636-1902      + | ENSCING00000017842
##   ENSCING00000016550        1  4219-20394      - | ENSCING00000016550
##   ENSCING00000016539        1 23518-34155      + | ENSCING00000016539
##   ENSCING00000002700        1 37108-60412      - | ENSCING00000002700
##   ENSCING00000002712        1 62406-66903      - | ENSCING00000002712
##                  ...      ...         ...    ... .                ...
##   ENSCING00000025225       MT 11275-11335      + | ENSCING00000025225
##   ENSCING00000025226       MT 11337-12899      + | ENSCING00000025226
##   ENSCING00000025227       MT 12910-12973      + | ENSCING00000025227
##   ENSCING00000025228       MT 12990-14724      + | ENSCING00000025228
##   ENSCING00000025229       MT 14725-14790      + | ENSCING00000025229
##                        gene_name   gene_biotype seq_coord_system
##                      <character>    <character>      <character>
##   ENSCING00000017842                   misc_RNA       chromosome
##   ENSCING00000016550             protein_coding       chromosome
##   ENSCING00000016539             protein_coding       chromosome
##   ENSCING00000002700             protein_coding       chromosome
##   ENSCING00000002712             protein_coding       chromosome
##                  ...         ...            ...              ...
##   ENSCING00000025225                    Mt_tRNA       chromosome
##   ENSCING00000025226         COI protein_coding       chromosome
##   ENSCING00000025227                    Mt_tRNA       chromosome
##   ENSCING00000025228       NADH5 protein_coding       chromosome
##   ENSCING00000025229                    Mt_tRNA       chromosome
##                                                                                              description
##                                                                                              <character>
##   ENSCING00000017842                                                                                NULL
##   ENSCING00000016550                                                                                NULL
##   ENSCING00000016539                                                                                NULL
##   ENSCING00000002700                                                                                NULL
##   ENSCING00000002712                                                                                NULL
##                  ...                                                                                 ...
##   ENSCING00000025225                                                                                NULL
##   ENSCING00000025226 cytochrome oxidase subunit I (mitochondrion)  [Source:RefSeq peptide;Acc:NP_758778]
##   ENSCING00000025227                                                                                NULL
##   ENSCING00000025228 NADH dehydrogenase subunit 5 (mitochondrion)  [Source:RefSeq peptide;Acc:NP_758779]
##   ENSCING00000025229                                                                                NULL
##                           symbol  entrezid
##                      <character>    <list>
##   ENSCING00000017842                    NA
##   ENSCING00000016550                    NA
##   ENSCING00000016539                    NA
##   ENSCING00000002700                    NA
##   ENSCING00000002712             100181294
##                  ...         ...       ...
##   ENSCING00000025225                    NA
##   ENSCING00000025226         COI    806118
##   ENSCING00000025227                    NA
##   ENSCING00000025228       NADH5    806127
##   ENSCING00000025229                    NA
##   -------
##   seqinfo: 763 sequences from KH genome

2 Creating EnsDbs for a given Ensembl release

This section describes the (semi)-automated way to create ensembldb EnsDb SQLite annotation databases using the Ensembl Perl API and the MySQL database dumps from Ensembl.

2.1 Requirements

2.2 Creating EnsDb SQLite databases

To create the databases we are sequentially downloading and installing the MySQL database dumps from Ensembl locally and query these local databases to extract the annotations that will be inserted into the final SQLite databases. While it would be possible to directly query the Ensembl databases using the Perl API, this has proven to be slower and frequently fails due to access shortages or time outs.

Below we load the ensembldb package and the generate-EnsDbs.R script in its scripts folder.

library(ensembldb)

scr <- system.file("scripts/generate-EnsDbs.R", package = "ensembldb")
source(scr)

We next use the createEnsDbForSpecies function. This function queries first the Ensembl ftp server to list all species for a certain Ensembl release. By default it will then process all species, by first downloading the MySQL database dumps, installing it into a local MySQL server and subsequently, using functionality from the ensembldb package and the Ensembl Perl API, exctracting all annotations from it and storing it to a SQLite datababse. The local MySQL database will then by default deleted (setting the parameter dropDb = FALSE does not delete the databases).

Important parameter to set in the function are: + ftp_folder: by default the main Ensembl ftp server will be queried (ftp://ftp.ensembl.org/pub), alternatively, also the Ensemblgenomes database can be queried. To build e.g. EnsDbs for all plants use ftp://ftp.ensemblgenomes.org/pub/release-34/plants/mysql. If this parameter is defined it has to point to the directory in which all MySQL database dumps can be found. + ens_version: the Ensembl version (e.g. 87). + user: the user name for the local MySQL database. + host: the hostname of the local MySQL database (e.g. "localhost"). + pass: the password for the local MySQL database.

Below we create all EnsDb databases for Ensembl release 87. Note that we have to have installed the Ensembl Perl API matching this release.

createEnsDbForSpecies(ens_version = 87, user = "someuser", host = "localhost",
                      pass = "somepass")

All SQLite database, one per species, will be stored in the current working directory.