New features and changes were introduced in IBM® InfoSphere™ Information Server, Version 8.5 along with documentation updates. The new and changed features and documentation updates are described in the following sections.
Table of contents
InfoSphere Information Server, Version 8.5, new features and changes:
Suite and product module changes
IBM InfoSphere Business Glossary
IBM InfoSphere Information Analyzer
IBM InfoSphere Information Services Director
IBM InfoSphere Metadata Workbench
Documentation changes included in the Version 8.5 release:
Suite and product module changes
- System requirements
- InfoSphere Information Server has added support for several new operating systems, database servers, application servers, and web browsers. For more information, refer to System Requirements.
- Installing
-
- Expanded high availability (HA) and scalability options
- InfoSphere Information Server now supports a wider variety of topologies for HA and scalability:
- The services tier can now be implemented within an IBM WebSphere® Application Server Network Deployment clustered environment.
- The metadata repository tier can now be implemented within a DB2® database cluster or HADR configuration, or an Oracle database Real Application Clusters (RAC) configuration.
- InfoSphere Information Server components can now be implemented on redundant servers that are managed by HA management software such as IBM Tivoli® System Automation for Multiplatforms.
- Redesigned installation program
- The InfoSphere Information Server installation program now runs within a standard Web browser, or in silent mode from the command-line. It performs a thorough requirements check that helps ensure that the installation is successful. It supports incremental removal of products and tiers as well as incremental installation. It can optionally install WebSphere Application Server and DB2. For more information, refer to Installing new software.
- Multiple analysis databases and how they relate to installing InfoSphere Information Server
- IBM InfoSphere Information Analyzer now supports multiple analysis databases. Each InfoSphere Information Analyzer project is associated with a specific analysis database. The analysis database might be used by a single InfoSphere Information Analyzer project, or it might be shared by multiple InfoSphere Information Analyzer projects. For more information about what you need to know at installation time, refer to Locations for the analysis databases.
- Secure HTTP (HTTPS) support
- IBM InfoSphere Information Server supports HTTPS communication between the application server and client components. If your environment requires confidentiality at this level, you can enable HTTPS.
- HTTPS provides two benefits:
- Encryption: Data sent over an HTTPS connection is encrypted.
- Identification: Communication is allowed only if the server is positively verified. Before communications begin, the server sends the client a certificate. The signature in the certificate is decrypted by the client to verify the authenticity of the sender.
- HTTPS configuration is optional. If you do not configure HTTPS, HTTP is used instead.
- For more information, refer to Secure Sockets Layer (SSL) configuration.
- Secure Sockets Layer (SSL) for inbound RMI/IIOP is disabled by default
- SSL for inbound RMI/IIOP is no longer enabled when you install InfoSphere Information Server. You must now actively enable this security feature. For more information, refer to Enablement of Secure Sockets Layer (SSL).
- Administering
-
- New Auditing Service
- The Auditing Service creates an audit trail of security-related events. This includes all activities that set or modify security-related settings and all user authentications and application logins. You can configure which audit events to log and how much information to include based on auditing requirements. For information about how to configure the Auditing Service, refer to Audit logging configuration.
- New roles
- There are new roles for common metadata, IBM InfoSphere Business Glossary, and InfoSphere Information Analyzer. For more information, refer to Security role overview.
- New entitlement scheme
- IBM has changed the entitlement scheme for InfoSphere Information Server. IBM no longer supplies each customer with a custom license file that programmatically restricts the product modules and components that can be installed and used. Instead, all InfoSphere Information Server product components and features are presented for installation. It is your responsibility to install only those items that are indicated on your Proof of Entitlement.
- When you install IBM InfoSphere DataStage®, the program prompts you to select the InfoSphere DataStage editions and feature packs to install and activate. Each item in the selection list enables associated InfoSphere DataStage canvases and job features. Select the items for which you have a valid Proof of Entitlement from IBM. The installation program activates the features that are associated with the items that you select. Any other editions or feature packs are deactivated and cannot be used.
- If you later acquire entitlements for an additional InfoSphere DataStage edition or feature pack, to use the features that are included in the item you must activate the item by using the LicensingServiceAdmin tool. If you no longer have entitlement for an item, you must deactivate it by using the tool. When you deactivate the edition or feature pack, the features within the item are no longer available for use.
- For more information, refer to Activating entitled IBM InfoSphere DataStage editions and feature packs.
- New URL for the web console
- The URL for the InfoSphere Information Server web console is http://hostname:9080/ibm/iis/console. The previous URL was http://hostname:9080/.
- Reporting
-
- Report templates
- Provides report templates for the suite components that you installed. The Information Center provides tables that detail the roles, permissions, and result formats for all report templates. For more information, refer to Available reports and templates.
- Data source connectors
-
- Connectors for Server jobs
- You can now use IBM InfoSphere Connectors to create and run InfoSphere DataStage Server jobs.
- Connector migration tool support
- Provides support for InfoSphere DataStage server jobs.
- z/OS® File Stage
- You can now use the Z/OS File Stage to read and write mainframe resident native files seamlessly from InfoSphere Information Server, running on non-Z/OS platforms. This requires IBM InfoSphere Classic Federation.
- Distributed Transaction Stage
- Provides the following additional targets: ODBC and IBM WebSphere MQ.
- Connectors support multiple input links
- You can now create multiple input links and configure the write methods in the target connector stage.
- New data integration project management tool
- Blueprint Director is a new capability to further strengthen the alignment of business and IT allowing you to govern your information projects. The integration team can collaborate on actionable information blueprints that connect the business vision with the corresponding technical metadata. It directs the team to apply best practices based on reference architectures and methodology.
- Moving assets between installations of InfoSphere Information Server
- You can now use the istool command line to move the following types of assets between metadata repositories:
- InfoSphere Business Glossary assets: Categories and terms and links to assigned assets and to stewards.
- InfoSphere FastTrack assets: Project templates, projects, mapping specifications, mapping components, and their related assets.
- InfoSphere Information Analyzer assets: Projects, data classes, and the common metadata and reports that are associated with projects.
- Reporting assets: Reports and report results that are created in the InfoSphere Information Server console or Web console.
- Security assets: Users, groups, roles, and credentials for InfoSphere Information Server.
- Full command-line support for MetaBroker® and bridge imports
- You can import metadata by using the command-line and automatically save your import parameters and filter settings to files for future command-line use. In previous releases, command-line support was limited and required manual creation of parameter XML files. For more information, refer to Importing metadata by using the command line.
IBM InfoSphere Business Glossary
- Enhanced InfoSphere Business Glossary Client for Eclipse integration with InfoSphere Data Architect
- InfoSphere Data Architect users can view updated content from the InfoSphere Business Glossary and create assigned relationships between these terms and data models in InfoSphere Data Architect. You can exchange these relationships between InfoSphere Data Architect and the business glossary. For more information, refer to Integration with Eclipse-based software.
- Enhanced integration with InfoSphere Information Analyzer: valid values
- InfoSphere Business Glossary users can view the valid values and valid values reference tables that were used as the basis of InfoSphere Information Analyzer analysis of database columns. Previous versions included other InfoSphere Information Analyzer analysis information but not valid values or valid values reference tables. For information about how to view the details of an asset, which includes these values, refer to Viewing the properties of an asset.
- Basic User role
- This new user role enables InfoSphere Business Glossary business users to view only the information most relevant to them. Users who are assigned only this InfoSphere Business Glossary role can view business terms and their attributes. These users cannot view assigned assets and data sources. For more information, refer to Roles.
- Business intelligence to business glossary import
- You can now import a business intelligence (BI) model, such as an IBM Cognos® Framework Manager model or a BusinessObjects Universe and create a structured glossary from it. Because, in many cases, business intelligence reports represent the intersection of business language and key data, these models can be excellent sources for initial population of glossary content. For more information, refer to Automatically generating IBM InfoSphere Business Glossary content from business intelligence models.
IBM InfoSphere DataStage
- High-availability configuration for the engine tier
- You can configure an active-passive configuration managed by high-availability cluster-management software, such as IBM Tivoli System Automation for Multiplatforms. For more information, refer to High availability configurations.
- Recovering from an engine tier failover
- New tools and features are available for ensuring the integrity of projects and data after an engine tier failover. For more information, refer to Engine tier failover recovery.
- Integrating with your source control system
- You can now use the InfoSphere Information Server Manager deployment tool to manage your InfoSphere DataStage and QualityStage jobs and associated assets in your source control system. The deployment tool gives you direct access to these source control systems: Concurrent Versions System (CVS) and IBM Rational® ClearCase®. You can use any source control system that supports the Eclipse Team programming interface. For more information, refer to Source control of InfoSphere DataStage and QualityStage assets.
- Optimizing parallel jobs in the DataStage Designer client
- Balanced Optimization is integrated into the Designer client to provide a more natural workflow for improving the performance of some IBM InfoSphere DataStage jobs. For more information, refer to Introduction to InfoSphere DataStage Balanced Optimization.
- Enhancements to the Transformer stage
- Null values can now be included in any expression, so you no longer need to explicitly handle null values in an expression. You can now define loops that output multiple output rows from every input row or that aggregate multiple input rows into a single input row. When your data is grouped on a column, and is sorted on that column, you can now detect when you process the last row before the value in that column changes. For more information, refer to Transformer stage.
- Vertical pivoting
- You can now specify a vertical pivoting operation in the Enterprise Pivot stage that maps a set of rows in the input data to single or multiple output columns. For more information, refer to Pivot Enterprise stage.
- Multiple null field values on Import
- You can now specify a set of values in the import data that are treated as null when importing data in stage such as the Sequential File stage. For more information, refer to Sequential File stage: Output link format tab.
- Copying roles from an existing project.
- You can now specify that a new project will inherit users, groups, and associated roles from an existing project. For more information, refer to Add Project window.
- Environment variables created during import
- When you import a job or parameter set that uses environment variables as parameters, if the environment variables do not already exist in the project, they will be created. for more information, refer to Importing previously exported objects.
- Missing columns in the Data Set stage
- You can now specify what action is taken for columns that are defined on the link but are not contained in the source data set. For more information, refer to Data Set stage: Options category.
- Saving job logs to the metadata repository
- When jobs run, you can save the logging information to the metadata repository of InfoSphere Information Server. For more information, refer to Enabling operational repository logging.
- Improved support features in the Designer client
- For more information, refer to Handling exceptions in the Designer client.
IBM InfoSphere FastTrack
- Data Rule Definitions
- You can use Data Rule Definitions (DRDs) to define the relationships between data sources when creating mapping specifications. For more information, refer to Creating shared data rules.
- Audit Trail history
- You can track the history of changes made in InfoSphere FastTrack for auditing purposes. For more information, refer to Tracking changes for auditing.
- Mapping report enhancements
- For a large number of column mappings, you can now view the mapping report in a specific order, such as by user, by table, by status, or by date. For more information, refer to Reports for IBM InfoSphere FastTrack.
- Improved mapping specification retrieval capability
- You can now retrieve mapping specifications from an existing InfoSphere DataStage job into InfoSphere FastTrack as mapping specifications or mapping compositions. For more information, refer to Generating mapping specifications from IBM InfoSphere DataStage jobs.
- Mapping compositions
- A mapping composition is a new first order object that allows you to define a set of mapping specifications and their relationships. Previously, you could only define compositions real-time when generating jobs. Now you can name these mapping compositions and reuse them. For more information, refer to Populating mapping specifications with mapping components.
- Specification Change Propagation updates
- Use the InfoSphere FastTrack Generation Wizard to determine if a targeted, existing InfoSphere DataStage job can be annotated. InfoSphere FastTrack can now detect the delta between a current job generation selection and the history of the first job iteration. The delta is then propagated to the related InfoSphere DataStage job. For more information, refer to Specification change propagation.
- Project template support
- A Project Template is available to facilitate the sharing of settings and objects across projects. The Project Template has common settings and objects that can be used by multiple projects. For more information, refer to Using a project template.
- Reject log records for lookup definitions
- You can now choose to log the reject records in a sequential file stage by using the modified Lookup Definition feature in InfoSphere FastTrack. For more information, refer to Adding source columns to mapping specifications from lookup tables.
- Hyperlink enhancements
- URLs can be made into hyperlinks in fields that link to an external document. For more information, refer to Creating mapping specifications.
- Context assistance feature
- From the Expression Editor field, press <ctrl> + space to open a list of valid functions and fields from which you can select to populate the expression editor. For more information, refer to Applying rule expressions on columns to transform data.
- Scope feature
- Define the scope for your mapping specifications so that you can later select the metadata to use from a consolidated list. For more information, refer to Defining the scope for your mapping specifications.
- Enhanced CSV Import
- You can create a mapping specification from a CSV (Comma Separated Values) file by combining multiple columns in the input file to create one or more values, such as a multi-part name, for your mapping specification. For more information, refer to Import from CSV options.
IBM InfoSphere Information Analyzer
IBM InfoSphere Information Analyzer can help you quickly and easily understand your data by offering data quality assessment, flexible data rules design and analysis, and quality monitoring capabilities. These insights can help you derive more information from enterprise data to accelerate information-centric projects.
- Supports sharing of InfoSphere Information Analyzer source analysis results through InfoSphere Business Glossary
- In addition to the existing summary results, you can now gain better understanding and insight into the valid values of source columns that are represented by business terms. For more information, refer to Metadata management and Local and global variables.
- Expanded collaboration between business users and IT
- You now have an improved ability to link data sources directly to business terms in InfoSphere Information Analyzer and deliver broader understanding of data to business users through context-sensitive menu options, enhanced term browsers, and presentation of terms with standard data classifications. For more information, refer to Creating, editing, or deleting InfoSphere Business Glossary terms.
- Supports asset interchange for projects and analysis results
- You can now export InfoSphere Information Analyzer project information to other InfoSphere Information Server environments to maximize distribution of data analysis insights. For more information, refer to InfoSphere Information Analyzer assets.
- Supports import of published analysis results
- You can import the analysis results from other IBM InfoSphere Information Server products to view high-level analysis results within InfoSphere Information Analyzer. For more information, refer to Importing published analysis results.
- Supports native Oracle connectivity
- You can leverage native connectors when you run column analyses, key analyses, and data rules. Teradata, DB2, and Oracle connectors are supported in addition to ODBC connectors.
- Supports external retrieval of information via HTTP API and HTTPS protocol
- You can now develop applications with the Hypertext Transfer Protocol (HTTP) application programming interface (API) to access and analyze InfoSphere Information Analyzer content. You can:
- Integrate InfoSphere Information Analyzer content in other non-IBM environments without the use of the rich client. For example, you can:
- Run column analysis or data rules from a web portal and retrieve the results
- Program the batch execution of a large set of rules and column analysis tasks with a non-IBM scheduler
- Generate custom reports, such as combine the results of several requests in one report and use XSLTs to format it as needed
- Deploy InfoSphere Information Analyzer project templates without the use of the rich client. For example, you can:
- Extract existing projects as XML documents
- Edit the XML manually if necessary and use source control to track changes
- Deploy on different systems. The rules are automatically recreated.
- Integrate InfoSphere Information Analyzer content in other non-IBM environments without the use of the rich client. For example, you can:
IBM InfoSphere Information Services Director
- New command line interface tool
- The InfoSphere Information Services Director administration and deployment command line interface can be used to administer InfoSphere Information Services Director resources such as applications and services. You can use this tool to move these resources between different installations of InfoSphere Information Services Director, deploy and undeploy these resources, and modify the metadata information in these resources. For more information, refer to IBM InfoSphere Information Services Director command line interface.
- New service bindings
- The Text over HTTP binding allows you to customize the format of messages for HTTP requests and response payloads to conform to specific requirements. When you use the Text over HTTP binding, you can send data in SOAP, XML, text, or any other format to InfoSphere Information Services Director by transforming the request payload. You can apply your own XSLT stylesheet before it is sent to the information provider to transform the request payload. The XSLT style sheet that you provide is also used to transform the information provider results before it is returned in the HTTP response. For more information, refer to Text over HTTP service binding.
- For distributed hypermedia systems, IBM InfoSphere Information Services Director can generate services that use a REST (Representational State Transfer) invocation: HTTP GET, POST, PUT, and DELETE for service requests and XML or JSON format for the response. For more information, refer to REST 2.0 service binding.
IBM InfoSphere Metadata Workbench
- Supports InfoSphere FastTrack mapping specifications for extended data lineage
- InfoSphere FastTrack allows the documentation of source-to-target mappings, which are similar to extended mapping documents in the metadata workbench. Business users can directly use InfoSphere FastTrack mapping specifications for extended data lineage without the need to import the mapping specification. For more information, refer to Configuring InfoSphere FastTrack mapping specifications for lineage reports.
- Supports IBM InfoSphere Discovery mapping specifications for extended data lineage
- InfoSphere Discovery allows the documentation of source-to-target mappings, which are similar to extended mapping documents in the metadata workbench. Business users can import InfoSphere Discovery mapping specifications for extended data lineage by using a configuration file in an XML format. The configuration file defines which InfoSphere Discovery column corresponds to which column the metadata workbench. For more information, refer to Importing extension mapping documents and their mappings.
- Business Intelligence assets can be deleted using the Repository Management tab of the Web console
- Business Intelligence (BI) assets can no longer be deleted in the Metadata Workbench. This functionality is available in the Repository Management tab of the Web console, where all types of common metadata can be browsed and deleted. For more information, refer to Managing metadata assets.
IBM InfoSphere QualityStage
- Combinable processing
- The Designer client now combines the process that executes in the Standardize stage with other operators in a job so that you have one process rather than a series of processes linked. You can reduce the footprint of the job by minimizing the number of processes that have to be started. Multi-national Standardize (MNS) reaps benefits from the combinable Standardize operator. For more information, refer to Advanced tab.
- Locale tokenization
- You can specify the regional setting that you want to use in a rule set, and thereby indicate the way you want tokens handled. For more information, refer to Specifying the tokenizer.
- Reporting for data standardization
- You can use the reports to assess the results of your standardization process and verify that the results meet your data quality objectives. For more information, refer to Standardization Quality Assessment (SQA) reports.
- Additional rules
- Rules enforce consistency and quality on the data that they are applied against. You now have an increased number of rule sets to handle country or region-specific data. For the list of predefined rule sets, refer to Predefined rule sets. For information on how to install rules from asset interchange, refer to Importing rule sets.
- Additional rules tools
- You can install a rule set development package from asset interchange. The package includes Standardization rules templates for domains, a Standardization rules development kit, and a Standardization quality assessment kit. For more information on the package, refer to Predefined rule sets.
- Enhanced Match Designer
- The content, format, and functionality within the Match Designer is enhanced for ease of use and provides more insight into the matching process. For example, you can view all match types and reorder Blocking Columns and Match Commands. For more information on creating match passes, refer to Defining match passes.
- Match performance improvements
- You can make changes to the Sort stage in the InfoSphere DataStage and QualityStage Designer client to improve match performance.
- Easily consumable reports for match specifications
- Use the reports to obtain sign off from business stakeholders on the criteria used to match data. For more information, refer to Reporting for match specifications.
Documentation introduced with Version 8.5
The following documentation additions and changes were introduced with InfoSphere Information Server Version 8.5.
- Reorganization of the Planning, Installation, and Configuration information
- The Installation information has been reorganized into the following sections so that it is easier for you to use:
- Planning for the installation of IBM InfoSphere Information Server
- Preparing the tier computers
- Installing IBM InfoSphere Information Server software
- Configuring the software after installation
- Creating highly available, scalable configurations
- Removing IBM InfoSphere Information Server software
- Troubleshooting installations
- The installation information now also includes checklists that guide the user through the installation of InfoSphere Information Server in various server topologies and configurations.
- New troubleshooting topics
- Added new troubleshooting topics for:
- General installation
- Authentication
- Application server issues
- Reporting
- Client and console
- Software removal problems
- Overview of IBM InfoSphere Information Server
-
- IBM InfoSphere Foundation Tools
- Added an overview of the IBM InfoSphere Foundation Tools. For more information, refer to InfoSphere Foundation Tools.
- Overview of high availability and clustering
- Added an overview of high availability and clustering. For more information, refer to Scalability of the common services tier and High availability in InfoSphere Information Server.
- New Auditing Service
- The Auditing Service creates an audit trail of security-related events. This includes all activities that set or modify security-related settings and all user authentications and application logins. You can configure which audit events to log and how much information to include based on auditing requirements. For an overview of the new Auditing Service, refer to Administrative services.
- IBM InfoSphere Classic Federation
- Added an overview of IBM InfoSphere Classic. For more information, refer to Classic products overview.
- Integration scenarios
- The integration scenarios focus on data quality within a data warehouse implementation. For more information, refer to InfoSphere Information Server integration scenarios.
- InfoSphere Information Analyzer methodology and best practices
- Includes a new section of methodology and best practices for working with rules in InfoSphere Information Analyzer by using defined and documented methods. For more information, refer to Data quality methodology.
- InfoSphere QualityStage
-
- Easily available pattern-action language information
- Now integrated into the Information Center. See IBM InfoSphere QualityStage Pattern Action reference.
- Major changes to the user's guide
- Includes a walk through of the overall InfoSphere QualityStage methodology, new information about using match weight overrides, and elaboration on concepts such as match. For more information, refer to InfoSphere QualityStage methodology and the topics that immediately follow the topic. For information on match weight overrides, refer to Specifying weight overrides. For information that explains key concepts of match, refer to the topics such as Match: key concepts, How match manages large volumes of data, and Weights and record comparisons.
- New information about designing real-time jobs
- Enhanced the InfoSphere Information Services Director user's guide. See Designing IBM InfoSphere DataStage and QualityStage jobs as services.
- Updated information about rule sets
- Included a comprehensive listing of predefined rule sets and rule set characteristics. For the list of predefined rule sets, refer to Predefined rule sets. For rule set characteristics, refer to the topics under Categories of rule sets. For characteristics of Japanese-specific rule sets, refer to Characteristics of Japanese rule sets.
9 comments:
Hi buddy great stuff....!!
As this is a huge content page. Scrolling throughout the page using scroll bar is not a good idea. My intention here is, if you can provide content links [e.g.: top, bottom] that could helps a lot..!!!
Hi friend, Its nice to see a comment from you. I have provided the expected links on this page. Also I'll provide the same links to all the pages as per the content requirement.
I request all my readers, If you have any questions or comments, or would like to share information always welcome and I'll do my best to answer the questions and to post reader's information.
Thanks n Regards
Venkat ♥ Duvvuri
http://datastage-venkat.blogspot.com/
Hi venkat i am very very much impressed to see your blog.
i got lot of information about datastage in this blog.
Hi Friend Thanks a lot for your comment on the same..
Thanks n Regards
Venkat ♥ Duvvuri
http://datastage-venkat.blogspot.com/
I would really like your post ,it would really explain each and every point clearly well thanks for sharing.
Hi Venkat,
Really i learnt a lot in your blog.
also i need one info on Datastage 8.5 is (approx 40%) faster when compare to earlier version. So do you know what all the changes they made to bring up the performance. (such as starting DataStage, opening a job, running a Parallel job and run-time performance)
Vadi
Hello Vadi,
It's nice to see your comment on my blog. Actually right now I m working on DS v8.1, So exactly I do not have any practice with DS v8.5. I'll try to find out the same as soon as possible. My Intention here is I would like to post the same with some prior practice especially performance related things.
Thanks n Regards
Venkat ♥ Duvvuri
http://datastage-venkat.blogspot.com/
Great to here this changes. I think this will be more useful for all of us.
Post a Comment