Contents

1 Introduction

Detailed billing information is essential for cost and resource planning in cloud based analysis projects, but it can be difficult to obtain. The goal of this software is to help users of the Terra/AnVIL platform get access to this data as easily as possible.

The google cloud platform console can be used to acquire information at varying levels of detail. For example, it is simple to generate a display like the following.

Cost track from Google console.

However, the cost track here sums up charges for various activities related to CPU usage, storage, and network use. Our aim is to provide R functions to help present information on charges arising in the use of AnVIL.

To whet the appetite, we will show how to run an exploratory app that looks like:

Early view of reckoning app.

2 Installation

Install the AnVILBilling package with

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager", repos = "https://cran.r-project.org")
BiocManager::install("AnVILBilling")

Once installed, load the package with

library(AnVILBilling)

3 Setup

The functions in this vignette require a user to connect the billing export in the Google Cloud Platform project associated with Terra/AnVIL to a BigQuery dataset.

General information on this process can be found here:

https://cloud.google.com/billing/docs/how-to/export-data-bigquery

In order to set this up with the AnVIL/Terra system:

  1. Create a new project in Google under the billing account linked to the Terra project
  2. Create a BigQuery dataset to store the billing export
  3. Go to the billing tab in the gcloud console and enable export to this dataset
  4. Make sure the user of this software has the BigQuery scope on the billing project

Once this is accomplished you will be able to see

exportview

with values appropriate to your project and account configuration substituted for ‘landmarkMark2’ (the compute project name), ‘bjbilling’ (the Google project with BigQuery scope that is used to transmit cost data on landmarkMark2 to Bigquery), and ‘anvilbilling’ (the BigQuery dataset name where next-day cost values are stored).

4 Obtaining billing data

4.1 Overview

Billing data is generally available 1 day after incurring charges. Billing data is stored in BigQuery in a partitioned table, and is queryable using the bigrquery package.

4.2 Setting up a request

In order to generate a request you need:

  1. start: date of start of reckoning
  2. end: date of end of reckoning
  3. project: GCP project id
  4. dataset: GCP dataset id for billing data in BigQuery
  5. table: GCP table for billing data in BigQuery

Then you can use the function:

setup_billing_request(start, end, project, dataset, table, billing_code)

To create a request.

Once you have a request object, then you can get the billing data associated with that request using the reckon() function on your billing request.

4.3 Output

The result of a reckoning on a billing request is an instance of avReckoning

We took a snapshot of usage in a project we work on, and it is available as demo_rec. This request represents one day of usage in AnVIL/Terra.

suppressPackageStartupMessages({
library(AnVILBilling)
library(dplyr)
library(magrittr)
library(BiocStyle)
})

demo_rec
## AnVIL reckoning info for project  bjbilling 
##   starting 2020-01-28, ending 2020-01-29.
## There are  1599  records.
## Available keys:
##  [1] "goog-resource-type"               "goog-metric-domain"              
##  [3] "goog-dataproc-cluster-name"       "goog-dataproc-cluster-uuid"      
##  [5] "goog-dataproc-location"           "cromwell-workflow-id"            
##  [7] "goog-pipelines-worker"            "terra-submission-id"             
##  [9] "wdl-task-name"                    "security"                        
## [11] "ad-anvil_devs"                    "ad-auth_anvil_anvil_gtex_v8_hg38"
## --- 
## Use ab_reckoning() for full table

The available keys for the billing object are shown.

For Terra, 3 of the most useful keys are:

  1. terra-submission-id : this key is associated with a cromwell workflow execution
  2. cromwell-workflow-id : this key is associated with a job in a workflow, so if you apply a workflow to multiple inputs, you will have 1 terra id and multiple cromwell ids
  3. goog-dataproc-cluster-name : this key is associated with a jupyter notebook or Rstudio cluster on Terra. The user of these resources can be determined using Bioconductor’s AnVIL package.

The code, to be used while the cluster is in use, would look like this:

library(AnVIL)
leo = Leonardo()
leo$getRuntime(clustername)

5 Drilling down

Given a key type, we want to know associated values.

v = getValues(demo_rec@reckoning, "terra-submission-id")
v
## [1] "terra-196d3163-4eef-46e8-a7e6-e71c0012003d"

To understand activities associated with this submission, we subset the table.

s = subsetByKeyValue(demo_rec@reckoning, "terra-submission-id", v)
s
## # A tibble: 955 × 17
##    billing_a…¹ service      sku          usage_start_time    usage_end_time     
##    <chr>       <list>       <list>       <dttm>              <dttm>             
##  1 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
##  2 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
##  3 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
##  4 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
##  5 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
##  6 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
##  7 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
##  8 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
##  9 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
## 10 015E39-385… <named list> <named list> 2020-01-28 20:00:00 2020-01-28 21:00:00
## # … with 945 more rows, 12 more variables: project <list>, labels <list>,
## #   system_labels <list>, location <list>, export_time <dttm>, cost <dbl>,
## #   currency <chr>, currency_conversion_rate <dbl>, usage <list>,
## #   credits <list>, invoice <list>, cost_type <chr>, and abbreviated variable
## #   name ¹​billing_account_id

The following data is available in this object

names(s)
##  [1] "billing_account_id"       "service"                 
##  [3] "sku"                      "usage_start_time"        
##  [5] "usage_end_time"           "project"                 
##  [7] "labels"                   "system_labels"           
##  [9] "location"                 "export_time"             
## [11] "cost"                     "currency"                
## [13] "currency_conversion_rate" "usage"                   
## [15] "credits"                  "invoice"                 
## [17] "cost_type"

You can drill down more to see what products used during the submission:

AnVILBilling:::getSkus(s)
##  [1] "Storage PD Capacity"                                                   
##  [2] "SSD backed PD Capacity"                                                
##  [3] "Network Inter Zone Ingress"                                            
##  [4] "Network Intra Zone Ingress"                                            
##  [5] "External IP Charge on a Standard VM"                                   
##  [6] "Custom Instance Ram running in Americas"                               
##  [7] "Custom Instance Core running in Americas"                              
##  [8] "Licensing Fee for Shielded COS (CPU cost)"                             
##  [9] "Licensing Fee for Shielded COS (RAM cost)"                             
## [10] "Network Internet Ingress from APAC to Americas"                        
## [11] "Network Internet Ingress from EMEA to Americas"                        
## [12] "Network Google Egress from Americas to Americas"                       
## [13] "Network Internet Ingress from China to Americas"                       
## [14] "Network Google Ingress from Americas to Americas"                      
## [15] "Network Internet Egress from Americas to Americas"                     
## [16] "Network Internet Ingress from Americas to Americas"                    
## [17] "Network Internet Ingress from Australia to Americas"                   
## [18] "Network HTTP Load Balancing Ingress from Load Balancer"                
## [19] "Network Inter Region Ingress from Americas to Americas"                
## [20] "Network Egress via Carrier Peering Network - Americas Based"           
## [21] "Network Ingress via Carrier Peering Network - Americas Based"          
## [22] "Licensing Fee for Container-Optimized OS from Google (CPU cost)"       
## [23] "Licensing Fee for Container-Optimized OS from Google (RAM cost)"       
## [24] "Licensing Fee for Container-Optimized OS - PCID Whitelisted (CPU cost)"
## [25] "Licensing Fee for Container-Optimized OS - PCID Whitelisted (RAM cost)"

You can also get the cost for a workflow using:

data(demo_rec) # makes rec
v = getValues(demo_rec@reckoning, "terra-submission-id")[1] # for instance
getSubmissionCost(demo_rec@reckoning,v)
## [1] 0.054044

And the ram usage as well:

data(demo_rec) # makes rec
v = getValues(demo_rec@reckoning, "terra-submission-id")[1] # for instance
getSubmissionRam(demo_rec@reckoning,v)
##                                 submissionID      workflow
## 1 terra-196d3163-4eef-46e8-a7e6-e71c0012003d runterratrial
## 2 terra-196d3163-4eef-46e8-a7e6-e71c0012003d runterratrial
## 3 terra-196d3163-4eef-46e8-a7e6-e71c0012003d runterratrial
## 4 terra-196d3163-4eef-46e8-a7e6-e71c0012003d runterratrial
## 5 terra-196d3163-4eef-46e8-a7e6-e71c0012003d runterratrial
##                                      cromwellID
## 1 cromwell-4dde8ce1-a8e5-47ba-a261-120ae8c7556c
## 2 cromwell-8edb23d6-a7d2-4b1f-96b4-496e96c1d707
## 3 cromwell-b1dcdbe1-b4ec-4428-8570-e2ab883087d0
## 4 cromwell-664b3519-7c7a-42ba-bb00-562ea1f650fd
## 5 cromwell-176a9f09-483c-4eb1-abf0-e5de2fdf36a7
##                                       sku       amount         unit
## 1 Custom Instance Ram running in Americas 1.676111e+12 byte-seconds
## 2 Custom Instance Ram running in Americas 1.737314e+12 byte-seconds
## 3 Custom Instance Ram running in Americas 1.607392e+12 byte-seconds
## 4 Custom Instance Ram running in Americas 1.573032e+12 byte-seconds
## 5 Custom Instance Ram running in Americas 1.582695e+12 byte-seconds
##     pricingUnit amountInPricingUnit
## 1 gibibyte hour           0.4336111
## 2 gibibyte hour           0.4494444
## 3 gibibyte hour           0.4158333
## 4 gibibyte hour           0.4069444
## 5 gibibyte hour           0.4094444

6 Using the exploratory app

To simplify some of the aspects of reporting on costs, we have introduced browse_reck, which will authenticate the user to Google BigQuery, and use user-specified inputs to identify an interval of days between which usage data are sought. This function can be called with no arguments, or you can supply the email address for the Google identity to be used in working with Google Cloud Platform projects and BigQuery.

7 Session Information

sessionInfo()
## R Under development (unstable) (2022-10-25 r83175)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.1 LTS
## 
## Matrix products: default
## BLAS:   /home/biocbuild/bbs-3.17-bioc/R/lib/libRblas.so
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_GB              LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] magrittr_2.0.3     dplyr_1.0.10       AnVILBilling_1.9.0 BiocStyle_2.27.0  
## 
## loaded via a namespace (and not attached):
##  [1] shinytoastr_2.1.1   tidyr_1.2.1         plotly_4.10.0      
##  [4] sass_0.4.2          utf8_1.2.2          generics_0.1.3     
##  [7] stringi_1.7.8       digest_0.6.30       grid_4.3.0         
## [10] evaluate_0.17       bookdown_0.29       fastmap_1.1.0      
## [13] jsonlite_1.8.3      DBI_1.1.3           promises_1.2.0.1   
## [16] BiocManager_1.30.19 httr_1.4.4          purrr_0.3.5        
## [19] fansi_1.0.3         viridisLite_0.4.1   scales_1.2.1       
## [22] lazyeval_0.2.2      jquerylib_0.1.4     cli_3.4.1          
## [25] shiny_1.7.3         rlang_1.0.6         dbplyr_2.2.1       
## [28] munsell_0.5.0       ellipsis_0.3.2      bit64_4.0.5        
## [31] cachem_1.0.6        yaml_2.3.6          tools_4.3.0        
## [34] gargle_1.2.1        colorspace_2.0-3    ggplot2_3.3.6      
## [37] httpuv_1.6.6        DT_0.26             assertthat_0.2.1   
## [40] vctrs_0.5.0         R6_2.5.1            mime_0.12          
## [43] lubridate_1.8.0     lifecycle_1.0.3     stringr_1.4.1      
## [46] fs_1.5.2            htmlwidgets_1.5.4   bit_4.0.4          
## [49] pkgconfig_2.0.3     gtable_0.3.1        pillar_1.8.1       
## [52] bslib_0.4.0         later_1.3.0         data.table_1.14.4  
## [55] glue_1.6.2          Rcpp_1.0.9          xfun_0.34          
## [58] tibble_3.1.8        tidyselect_1.2.0    knitr_1.40         
## [61] xtable_1.8-4        htmltools_0.5.3     rmarkdown_2.17     
## [64] compiler_4.3.0      bigrquery_1.4.1