error saving file!error saving file!error saving file!error saving file! June, 2021 - Relevance Lab

Your address will show here +12 34 56 78

Relevance Lab and AWS collaborate to release new EC2-RStudio-Server solution for Service Workbench to streamline scientific research tools in the cloud. Relevance Lab provides a secure, scalable, and cost-effective solution for RStudio usage. The open source contributions will create multiple solutions for frictionless Scientific Research on AWS.

Click here
 for the full  story.

0

2021 Blog, Blog, Featured, SWB Blog

Provide researchers access to secure RStudio instances in the AWS cloud by using Amazon issued certificates in AWS Certificate Manager (ACM) and an Application Load Balancer (ALB)

Cloud computing offers the research community access to vast amounts of computational power, storage, specialized data tools, and public data sets, collectively referred to as Research IT, with the added benefit of paying only for what is used. However, researchers may not be experts in using the AWS Console to provision these services in the right way. This is where software solutions like Service Workbench on AWS (SWB) make it possible to deliver scientific research computing resources in a secure and easily accessible manner.

RStudio is a popular software used by the Scientific Research Community and supported by Service Workbench. Researchers use RStudio very commonly in their day-to-day efforts. While RStudio is a popular product, the process of installing RStudio securely on AWS Cloud and using it in a cost-effective manner is a non-trivial task, especially for Researchers. With SWB, the goal is to make this process very simple, secure, and cost-effective for Researchers so that they can focus on “Science” and not “Servers” thereby increasing their productivity.

Relevance Lab (RL), in partnership with AWS, set out to make the experience of using RStudio with Service Workbench on AWS simple and secure.

Technical Solution Goals

  1. A researcher should be able to launch an RStudio instance in the AWS cloud from within the Service Workbench portal.
  2. The RStudio instance comes fully loaded with the latest version of RStudio and a variety of other software packages that help in scientific research computing.
  3. The user launches a URL to the RStudio from within the Service Workbench. This URL is a unique URL generated by SWB and is encoded with an authentication token that ensures that the researcher can access the RStudio instance without remembering any passwords. The URL is served over SSL so that all communications can be encrypted in transit.
  4. Maintaining the certificates used for SSL communication should be cost-effective and should not require excessive administrative efforts.
  5. The solution should provide isolation of researcher-specific instances using allowed IP lists controlled by the end-user.

Comparison of Old and New Design Principles to make Researcher Experience Frictionless
The following section summarizes the old design and the new architecture to make the entire researcher experience frictionless. Based on feedback from researchers, it was felt that the older design required a lot of setup complexity and lifecycle upgrades for security certificate management, slowing down researchers productivity. The new solution makes the lifecycle simple and frictionless along with smart and innovative features to keep ongoing costs optimized.


No. RStudio Feature Original Design Approach New Design Approach
1 User Generated Security Certificate for SSL Secure Connections to RStudio. Users have to create a certificate (like LetsEncrypt) and use it with RStudio EC2 Instance with NGINX server. This creates complexity in the Certificate lifecycle. Complex for end-users to create, maintain and renew. The RStudio AMI also needs to manage the Certificate lifecycle. Move from External certificates to AWS ACM.

Bring in a shared AWS ALB (Application Load Balancer) and use AWS ACM certificates for each Hosting Account to simplify the Certificate Management Lifecycle.
2 SSL Secure Connection. Create an SSL connection with Nginx Server on RStudio EC2. Related to custom certificate management. Replaced with ALB at an Account level and shared by all RStudio Instances in an account. User Portal to ALB connection secured by ACM. For ALB to RStudio EC2 secure connection, use unique self-signed Certificates to encrypt connection per RStudio.
3 Client Role (IAM) changes in SWB. Client role is provided necessary permissions for setup purposes. Additional role privileges added to handle ALB.
4 ALB Design. Not existing in the original design. Shared ALB design per Hosting Account to be shared between Projects. Each ALB is expected to cost about $20-50 monthly in shared mode with average use. API model used to create/delete ALB.
5 Route 53 Changes on the Main account. A CNAME record gets created with the EC2 DNS name. A CNAME record gets created with the ALB DNS name.
6 RStudio AMI. Embedded with Certificate details. Related to custom certificate management. Independent of user-provided Certificate details. Also, AMI has been enhanced to include the following: Self-signed SSL and additional packages (as commonly requested by researchers) are baked into the AMI.
7 RStudio Cloud Formation Template (CFT). Original one to be removed from SWB. Added a new output to indicate the “Need ALB” flag. Also, create a new target group to which the ALB can route requests.
8 SWB Hosting Account Configuration. Did not have to provision certificate AWS ACM. Manual process to set up a certificate in a new hosting account.
9 Provisioned RStudio per Hosting Account Active Count Tracking. None. Needed to ensure ALB is created the first time when RStudio is provisioned and deleted after the last RStudio is deleted to optimize cost overheads of ALB.
10 SWB DynamoDB Table Changes. DynamoDB used for all Tables by SWB. Modifications needed to support the new design. Added to the existing DeploymentStore table in SWB design.
11 SWB Provision Environment Workflow. Standard design. Additional Step added to check if “Workspace Type” needs ALB and if it does when checking for ALB and either create or pass the reference to existing one.
12 SWB Terminate Environment Workflow. Standard design. Additional Step added to check if last Active RStudio being deleted and if so, also delete ALB to reduce idle costs.
13 Secure “Connection” Action from SWB Portal to RStudio instance. To ensure each RStudio has a secure connection for each user a unique connection URL is generated during the user session that is valid for a limited period. The same design of the original implementation is preserved. Internally the routing is managed through ALB but the concept remains the same. This ensures users do not have to remember user id/password for RStudio and a secure connection is always made available.
14 Secure “Connection” from SWB Portal disallowing other users from accessing RStudio resources given shared ALB. NA. Using the design feature (Step-13) ensures that even post ALB the connection for a User (Researcher and PI) is still restricted to their provisioned RStudio only and they cannot access other Researchers Instances. The unique connection is system generated using User to RStudio mapping uniquely.
15 ALB Routing Rules for RStudio secure connections given shared nature. NA. Every time an RStudio is created or deleted, changes are made to ALB rules to allow a secure connection between the User session and the linked RStudio. The same rules are cleaned up during RStudio delete lifecycle. These changes to ALB routing rules are managed from SWB code under Workflow customizations. (Step-11 and 12) using APIs.
16 RStudio Configuration parameters related to CIDR. Original design allows only whitelisted IP addresses to connect to associated RStudio instances – this can be modified also from configurations. RStudio Cloud Formation Template (CFT) should take Classless Inter-Domain Routing (CIDR) as Input Parameter and pass it through as an Output Parameter for the SWB to take it and create the ALB Listener Rule.
SWB code will take CIDR from RStudio CFT output, subsequently, update the ALB Listener Rule with the respective Target Group.
17 Researcher costs tracking. The original design had RStudio costs tracked for Researchers. Custom certificate costs were not tracked if any. In the new design, RStudio costs are tagged and tracked per researcher. ALB costs are treated as shared costs for the Hosting account.
18 RStudio Packaging and Delivery for a new customer – Repository Model. Bundled with standard SWB repo and installed. New model for RL to create a separate Repo and host RStudio with associated documentation and templates for customers to use.
19 RStudio Packaging and Delivery for a new customer – AWS Marketplace model. None. RL to provide RStudio on AWS Marketplace for SWB customers to add to standard Service Catalog and import (Future Roadmap item).
20 Upgrade and Support Models for RStudio. SWB teams ownership. To be managed by RL teams.
21 UI Modification for Partner Provided Products. No partner provided products. Partner-provided products will reside in the self-hosted repo. SWB UI will provide a mechanism to show details of partner names and a link to additional information.


The diagram below explains the interplay between different design components.


Secure and Scalable Solution Architecture
Keeping in mind the above design goals, a secure and scalable architecture is implemented that solves the problem of shared groups using products like RStudio requiring secure HTTPS access without the overheads of individual certificate management. The architecture also enables sharing the same concept for all future researcher products with similar needs without any additional implementation overheads resulting in increased productivity and lower costs.


The Relevance Lab team designed a solution centered on an EC2 Linux instance with RStudio and relevant packages pre-installed and delivered as an AMI.

  1. When the instance is provisioned, it is brought up without a public IP address.
  2. All traffic to this instance is delivered via an Application Load Balancer (ALB). The ALB is shared across multiple RStudio instances within the same account to spread the cost over a larger number of users.
  3. The ALB serves over an SSL link secured with an Amazon-issued certificate which is maintained by AWS Certificate Manager.
  4. The ALB costs are further brought down by provisioning it on demand when the first RStudio instance is provisioned. Conversely, the ALB is de-provisioned when the last RStudio instance is de-provisioned.
  5. Traffic between the ALB and the RStudio instance is also secured with an SSL certificate which is self-signed but unique to each instance.
  6. The ALB listener rules enforce the IP allowed list configured by the user.

Conclusion
Both SWB and Relevance Lab RLCatalyst Research Gateway teams are committed to making scientific research frictionless for researchers. With a shared goal, this new initiative speeds up collaboration and will help provide new innovative open-source solutions leveraging Service Workbench on AWS and partner-provided solutions like this RStudio with ALB from Relevance Lab. The collaboration efforts will soon be adding more solutions covering Genomic Pipeline Orchestration with Nextflow, use of HPC Parallel Cluster, and secure research workspaces with AppStream 2.0, so stay tuned.

To get started with RStudio on SWB provided by Relevance Lab use the following link:
Relevance Lab Github Repository for SWB Templates

For more information, feel free to contact marketing@relevancelab.com.

References
Service Workbench on AWS for driving Scientific Research
Service Workbench on AWS Documentation
Service Workbench on AWS Github Repository
RStudio Secure Architecture Patterns
Relevance Lab Research Gateway



0

2021 Blog, AppInsights Blog, Blog, Featured

Many AWS customers either integrate ServiceNow into their existing AWS services or set up both ServiceNow and AWS services for simultaneous use. Customers need a near real-time view of their infrastructure and applications spread across their distributed accounts.

Commonly referred to as the “Dynamic Application Configuration Management Database (CMDB) or Dynamic Assets” view, it allows customers to gain integrated visibility into their infrastructures to break down silos and facilitate better decision making. From an end-user perspective as well, there is a need for an “Application Centric” view rather than an “Infrastructure/Assets” view as better visibility ultimately enhances their experience.

An “Application Centric” View provides the following insights.

  • Application master for the enterprise
  • Application linked infrastructure currently deployed and in use
  • Cost allocation at application levels (useful for chargebacks)
  • Current health, issues, and vulnerability with application context for better management
  • Better aligned with existing enterprise context of business units, projects, costs codes for budget planning and tracking

Use Case benefits for ServiceNow customers
Near real-time view of AWS applications & Infrastructure workloads across multiple AWS accounts in ServiceNow. Customer is enabling self-service for their Managed Service Provider (MSP) and their Developers to:

  • Maintain established ITSM policies & processes
  • Enforce Consistency
  • Ensure Compliance
  • Ensure Security
  • Eliminate IAM access to underlying services

Use Case benefits for AWS customers
Enabling application self-service for general & technical Users. The customer would like service owners (e.g. HR, Finance, Security & Facilities) to view AWS infrastructure-enabled applications via self-service while ensuring:

  • Compliance
  • Security
  • Reduce application onboarding time
  • Optical consistency across all businesses

RLCatalyst AppInsights Solution – Built on AppRegistry
Working closely with AWS partnership groups in addressing the key needs of customers, RLCatalyst AppInsights Solution provides a “Dynamic CMDB” solution that is Application Centric with the following highlights:

  • Built on “AWS AppRegistry” and tightly integrated with AWS products
  • Combines information from the following Data Sources:
    • AWS AppRegistry
    • AWS Accounts
      • Design time Data (Definitions – Resources, Templates, Costs, Health, etc.)
      • Run time Data (Dynamic Information – Resources, Templates, Costs, Health, etc.)
    • AppInsights Additional Functionality
      • Service Registry Insights
      • Aggregated Data (Lake) with Dynamic CMDB/Asset View
      • UI Interaction Engine with appropriate backend logic


A well-defined Dynamic Application CMDB is mandatory in cloud infrastructure to track assets effectively and serves as the basis for effective Governance360.

AWS recently released a new feature called AppRegistry to help customers natively build an AWS resources inventory that has insights into uses across applications. AWS Service Catalog AppRegistry allows creating a repository of your applications and associated resources. Customers can define and manage their application metadata. This allows understanding the context of their applications and resources across their environments. These capabilities enable enterprise stakeholders to obtain the information they require for informed strategic and tactical decisions about cloud resources. Using AppRegisty as a base product, we have created a Dynamic Application CMDB solution AppInsights to benefit AWS and ServiceNow customers as explained in the figure below.



Modeling a common customer use case
Most customers have multiple applications deployed in different regions constituting sub-applications, underlying web services, and related infrastructure as explained in the figure below. The dynamic nature of cloud assets and automated provisioning with Infrastructure as a Code makes the discovery process and keeping CMDB up to date a non-trivial problem.



As explained above, a typical customer setup would consist of different business units deploying applications in different market regions across a complex and hybrid infrastructure. Most existing CMDB applications provide a static assets view that is incomplete and not well aligned to growing needs for real-time application-centric analysis, costs allocation, and application health insights. This problem has been solved by the AppInsights solution leveraging existing investments of customers on ITSM licenses of ServiceNow and pre-existing solutions from AWS like ServiceManagement connector that are available for no additional costs. The missing piece till recently was an Application-centric meta data linking applications to infrastructure templates.

Customers need to be able to see the information across their AWS accounts with details of Application, Infrastructure, and Costs in a simple and elegant manner, as shown below. The basic KPIs tracked in the dashboard are following:

  • Dashboard per AWS Account provided (later aggregated information across accounts to be also added)
  • Ability to track an Application View with Active Application Instances, AWS Active Resources and Associated Costs
  • Trend Charts for Application, Infrastructure and Cost Details
  • Drill-down ability to view all applications and associated active instances what are updated dynamically using a period sync option or on-demand use based

The ability to get a Dynamic Application CMDB is possible by leveraging the AWS Well Architected best practices of “Infrastructure as a Code” relying on AWS Service Catalog, AWS Service Management Connector, AWS CloudFormation Templates, AWS Costs & Budgets, AWS AppRegistry. The application is built as a scoped application inside ServiceNow and leverages the standard ITSM licenses making it easy for customers to adopt and share this solution to business users without the need for having AWS Console access.



Workflow steps for adoption of RLCatalyst AppInsights are explained below. The solution provided is based on standard AWS and ServiceNow products commonly use in enterprises and build on existing best practices, processes and collaboration models.


Step-1 Define AppRegistry Data Use AppRegistry
Step-2 Link App to Infra Templates – CloudFormation Template (CFT) / Service Catalog (SC) AWS Accounts Asset Definitions
Step-3 Ensure all Assets Provisioned have App and Service Tagging (Enforce with Guard Rails) AWS Accounts Asset Runtime Data
Step-4 Register Application Services – Service Registry Service Registry
Step-5 AppInsights Data Lake refresh with static and dynamic Updates (Aggregated across accounts) RLCatalyst AppInsights
Step-6 Asset, Cost, Health Dashboard RLCatalyst AppInsights


A typical implementation of RLCatalyst AppInsights can be rolled out for a new customer in 4-6 weeks and can provide significant business benefits for multiple groups enabling better Operations support, Self-service requests, application specific diagnostics, asset usage and cost management. The base solution is built on a flexible architecture allowing for more advanced customization to extend with real time health and vulnerability mappings and achieve AIOps maturity. In future there are plans to extend the Application Centric views to cover more granular “Services” tracking for support of Microservice architectures, container based deployments and integration with other PaaS/SaaS based Service integrations.

Summary
Cloud-based dynamic assets create great flexibility but add complexity for near real-time asset and CMDB tracking. While existing solutions using Discovery tools and Service Management connectors provided a partial solution to an Infrastructure centric view of CMDB, a robust Application Centric Dynamic CMDB was a missing solution that is now addressed with RLCatalyst AppInsights built on AppRegistry as explained in the above blog.

For more information, feel free to contact marketing@relevancelab.com.

References
Governance360 – Are you using your AWS Cloud “The Right Way”
ServiceNow CMDB
Increase application visibility and governance using AWS Service Catalog AppRegistry
AWS Security Governance for Enterprises “The Right Way”
Configuration Management in Cloud Environments



0