Can we create a complete Secure Central repository with Access restriction( Read) to the Developers and Full Access(Full) to the Admin??

Yes this is pretty much possible with the Secure Central Repository in BODS.

It pretty simple.

You will be having the bellow users created.

1) Admin

2) Developer 1, 2, .. n

Go to Management console --> Administration --> Central repository (you can see your secure central repositories available here).

Expand your central repo and you can See Users and Groups.

Create two Groups Eg:- Dev and Admin. You will have a default Group called DIGroup.

Now go to Users Tab and map your Admin to Admin Group and DI Group and Developers to only Dev Group.

By this you are restricting the Developers with read access and providing full permission to the Admin

Now Go to your designer and configure your CR and login with the Admin Credentials (username and pwd)

Go to your Data stores and do a Check out ( Objects & dependents )

The developers will configure the same CR and login to that with Developer credentials.

Now the Developers will have a read access and on top of that those Data stores are Checked out by the Admin.

This gives you a full Secure Central repository.

Rakesh

Although it is not very complex transform, but let us go somewhat deeper into the basics of it and find out the basic needs of this transform. and a view of operation-codes for beginners.

The purpose of Map_Operation transform is to map/change the Operation-Code of incoming/input row to the desired Operation-Code/Operation to be performed on the target table.

Why We need it After all? : It might be the case that the incoming/input row needs to be updated in the target table ( because some data has been changed since the target was populated last time) , but instead of updating it in the Target_table, you are willing to insert a new row and keep the older one as well OR you might be willing to delete the changed row from the target table OR you might be willing to do nothing at all for the rows which are changed. Hence, the operation-codes are the arms which help doing all these things.

But what are these Operation-Codes? : Let us suppose you have one source table named "source_table" and one target table named "target_table" . Both have exactly same schema. Now , in the first run you populate the target_table with all the data in source_table , hence both tables have got exactly the same data as well. Now some changes are done on the Source_table, few new rows are added , few rows got updated , few rows are deleted . Now if we compare the Source_table with the Target_table and try to set the status of each row in the Source_table relative to the Target_table then we'll have one of the following statuses for each input row :

-The rows which are there in the Source_table and not in the Target_table, basically the new rows: These rows need to be inserted to the Target_table, hence the Operation-Code "insert" will be associated with these rows coming from the Source_table.

-The rows found in both tables, but are updated in the Source_Table : These rows need to be updated in the Target_table, hence the Operation-Code  "update" will be associated with these rows coming from the Source_table.

-The rows which are there in Target_Table and deleted from Source_Table after the last run. These rows need to deleted from the Target_table (although we hardly perform deletion in a datawarehouse), hence the Operation-Code "delete" will be associated with each row of this kind.

-The rows which are there in both of the tables. These rows ideally doesn't need any operation , hence the Operation-Code "normal" is associated with such rows.

Well, how to perform this comparison of Source and Target Tables? : This can be done by the Table_Comparison transform. It compares the input table (Source_table in our example) with the another table called as comparison_table in BODS jargon (Target_table in our recent example). and after comparing each row it associates an Operation-Code to each row of the input table. and if we choose , it also detects the rows which were deleted from input table (to choose whether we need to perform deletion on Target_table or not). But we are not going in the details of Table_Comparison transform here, as i was going to play with the map_operation transform alone, and i know it looks crazy to do so. Because , like in the figure given below, if i connect the Source Table directly to the Map_Operation transform, "by default" the operation-code associated with all rows is "normal", until we change it using table_comparison transform.

Playing with Basics of Map_Operation Transform: So, as said earlier the job of map_operation transform is to change/map the incoming op-code (operation-code) to the desired/outgoing op-code. See the pic below we have options of "Input row type" and "Output row type". But, in the above mapping because we have connected the source directly to the map_operation transform, All Incoming Rows Have the Op-Code "normal" Associated with them. Hence, the second, third and fourth option doesn't matter for now because there are no input rows with operation-code "update" or "insert" or "delete" associated with them.

So, let us see what will happen if we use one by one the provided options in the drop-down menu given for the "Output row type". The interesting ones are "update" and "delete". let us see why:

First,  Let us suppose there is no primary-key defined on the target table :

-"normal" : If we choose "normal" as Output row type op-code, all input rows will be inserted to the target table.

-"insert" : same as "normal" , all input rows will be inserted to the target table.

-"discard": Nothing will be done.

-"delete": It'll delete the matching rows from the target table. Even if the primary key is not defined on target , delete operation is performed by matching the entire row. i.e. Value of each field in the input row matched with Value of each field in the Target table row, if matched the target row is deleted. No need of a Key column for delete operation.

-"update": This time, when no primary key is defined on the target , the update operation will not do anything. This is actually logical, I have an input row with a key field (say EmpNo) , but how can i ask my software to update the this row in target table if i haven't mentioned the same key in the target table as well? How my software will find the row which needs to be updated ? If i say , match the entire row , then it'll find the exact match of entire row if found, and that would mean that there is nothing to be updated as the entire row matches. So, i need to mention something (some column) in the target using which i can find the row to be updated.

So, there are two ways to do this:

First is to define a Primary key on the target table and re-import the table.

Second(assuming that input table has a primary key defined) is open the table by double-clicking on it, in the option tab we have the option to "Use Input keys" as shown below. By choosing yes here, It'll use the Primary key mentioned in the input table, provided that the column with same name and datatype is present in the Target table as well.

Secondly If a primary key is defined on the Target table as well then the "normal" and "insert" operations will fail if a row with same primary-key value will be tried to be inserted again, the job will fail and stop.Whereas the "update" and "delete" operations would work as before.

The behavior of map_operation transform changes somewhat when it is preceded by the table_comparison transform. The need of primary key on the target Or mentioning to use the input keys (as shown above) eliminates, It updates the target rows without it as well. Might be because we mention the primary key column of the input table in the table_comparison transform along with the comparison columns, and might be the rows carry this information along with the associated op-code to the map_operation transform. I checked and experimented this practically but can just guess about the internal engineering of it.

Has anyone successfully Consumed an authenticated Web Service in Data Services Designer from Net Weaver?

I just helped a customer install Data Quality Management for SAP from scratch.  2 days start-to-finish, including customizations.  We probably could have saved some time if we had all the files downloaded prior to starting the install.

Ingredients (all available from Software Download Center):

• IPS 4.0 SP4 (release notes)
• DS 4.1 SP1 Patch 1 (release notes)
• DQM 4.0 SP02 -- Data Services features -- e.g. the jobs themselves (release notes)
• DQM 4.0 SP02 -- Java features  (e.g. the RFC servers)
• DQM 4.0 SP02 -- ABAP features (440_731 SP02)
• an existing SAP ECC EHP6 system running on NW 7.31
• Data Quality address directories

Instructions:

• IPS install guide | admin guide
• DS install guide | admin guide
• DQM user's guide -- basically like an installation guide...very detailed and accurate! Read this several times before starting!

Most importantly, a list of SAP Notes we found helpful and got us past errors and hiccups.

1740516 - Compatibility requirements between DS, IPS, and Cleansing Package

1720236 - Available releases and patch levels of Data Services

1742633 - Permission error when logging into default local repo after DS 4.1 installation

1887978 - Address Cleanse Suggestions realtime job crashes with Access Violation

1732816 - How to manually import DQM ATL files correctly after failed install

1506464 - What version of DQM is deployed into SAP?

1857608 - BC set activation error on /FLDQ/RSPRODVER report

1517544 - How to activate or reactivate Business Configuration sets for DQM

1373324 - How long can expired USPS directories be used when running in noncertified mode?

1644004 - Enable or disable RFC Server trace logging (traces DQM traffic between SAP & DS)

1529071 - Sizing how many RFC Servers you should use

1544413 - Troubleshooting RFC Server - composite note

1764059 - Troubleshooting RFC Server - can't find log file

1851293 - Troubleshooting RFC Server - not all services found

Sure, this presumes you know how to install & configure a basic SAP Data Services environment. But it went surprisingly quick and worked basically out-of-the-box. And if you have errors and roadblocks, SAP is here to support you if you create a message to component EIM-DQM-SAP.

As part of our ASUG / TechEd "Meet the Speaker" blog series, I am honored to introduce ASUG TechEd speaker Emil Beloglavec of Kellogg's Company of Battle Creek, Michigan.  Emil is presenting at SAP TechEd Las Vegas in October.

I spoke with Emil last month and he is great at sharing his knowledge of SAP solutions and tools.  His session is RDP 143 Kellogg’s Data – How to Deliver One Version of the Truth?

Q: What will attendees learn at your session?

A: Kellogg is stepping up towards new technologies, for example: BW to HANA, CRM on HANA, BPC on HANA, and mining big data in form of structural (POS) or textual form (Hadoop). We strongly believe that first foundations have to be laid; one of them is definitely to achieve data quality and data harmonization across SAP and non-SAP systems. The presentation is a show case of Kellogg’s path taken so far and the possibilities ahead of us. Its main goal is to encourage a discussion in the spirit of knowledge exchange.

Q: How did you get your start with SAP?

A: Due to my previous experience with Data Warehousing ( Oracle and Cognos ), I started with SAP BW when I started with Kellogg about 8 years ago. For the past three years I work with SAP BusinessObjects Data Services.

Q: What is your favorite hobby?

A: Listening to classical music, especially to composers like Rachmaninov, Tchaikovsky, Chopin, Liszt, Saint-Saens, Grieg, Beethoven and the list could go on and on.

Here is hisAbstract:

Kellogg began building a new SAP ERP/SAP NetWeaver Business Warehouse environment, replacing one of its three SAP instances four years ago. SAP Data Insight and SAP Data Services were selected to assess the quality of data and convert data from legacy SAP/non-SAP systems into the new environment. This project is in its final stage of deployment and they are moving into sustainment phase. One of the driving factors for this project was to provide accurate data so that the business obtains the right answers at the right time. The presenters of this session will share lessons learned about assessing data, conversion, dual maintenance, generations of tools involved, a vision of the future, and how timely delivery of accurate data can support it.

Check out Emil's presentation from ASUG 2012 Annual Conference presentation slides - 2 Go-Lives 0 Data Defects with data migrations - that is a great achievement.

Add Emil's SAP TechEd session to your favorite list today and meet him in person in Las Vegas in October.

Related Links:

Beer and Analytics by Alexandre Papagiannidis Rivet

Workflow Approval Anywhere, Anytime - Meet the Speaker Graham Robinson - ASUG SAP TechEd session

Meet the ASUG TechEd Speaker Dennis Scoville - Empowered Self-Service BI with SAP HANA and SAP Lumira

Poll: What Version of Data Services are you using? by Ina Felsheim

Great TechEd pre-conference session: Before you Start, Make a Plan by Ina Felsheim

ASUG TechEd Pre-Conference Sessions

Hi ,

We have a scenario  that  need to extract the vendor data from People Soft to SAP by using BODS. We are using IDOC standard job to load into SAP.

In these loads, we have 2 fields Alternate Payee (LNRZB) and Fiscal Address Vendors ( FISKN) which should be updated with the SAP Vendor ids which have alternate payees and fiscal address respectively, but before loading into SAP we won't be having SAP IDs, so is there anyway we can load data for these 2 fields without running Job twice?? ( first time we load all the records into SAP by keeping these as null and then next time we have to update these fields)

In Informatica, we have dynamic lookup which gets data at run time into lookup table from target table, is there any such thing available in SAP BODS ??

Steps for executing BODS job from Unix Script

This will help you understand how to change the global parameters used in the job during execution of the job invoked via a Unix Script.

While you export the .SH for job execution, the default parameter values or last used parameter value will be attached within the .SH file. Whenever you execute that .SH file, the job starts with the same parameters all the time. You may need to modify the .SH file all the time, whenever you need to make changes in the user parameter (Global parameter) values. Go through the simple steps involved in resolving this issue effectivelty with minor modifications and a simple unix script to pass new user defined values and execution of the BODS job.

Log in to Data Service Management Console

Go to Administrator-> Batch (Choose Repository)

  Click on Batch Job Configuration tab to choose  the job which needs to be invoked through Unix

  Click on Export Execution Command Option against the job

Click on Export.

Two Files then will get generated and placed in the Unix Box. **

One .TXT file named as “Reponame.Txt” in /proj/sap/SBOP_INFO_PLAT_SVCS_40_LNX64/dataservices/conf

One .sh file named as “jobname.sh” in /proj/sap/SBOP_INFO_PLAT_SVCS_40_LNX64/dataservices/log

**Location will be changed according to the setup

1. For a job with no user entry parameters required, we can directly call the .sh file generated for job execution.

. ./Job_Name.sh

2. For a job which has parameters the script
will look like this

/proj/sap/SBOP_INFO_PLAT_SVCS_40_LNX64/dataservices/bin/AL_RWJobLauncher "/proj/sap/SBOP_INFO_PLAT_SVCS_40_LNX64/dataservices/log/DEV_JS_1/"
-w "inet:acas183.fmq.abcd.com:3500" " -PLocaleUTF8 -R\"REPO_NAME.txt\"  -G"1142378d_784a_45cd_94d7_4a8411a9441b"
-r1000 -T14 -LocaleGV -GV\"\$Character_123=MqreatwvssQ;\$Integer_One=Qdasgsssrdd;\"  -GV\"DMuMDEn;\"    -CtBatch -Cmacas183.fmq.abcd.com -CaAdministrator -Cjacas183.fmq.abcd.com -Cp3500 "

The highlighted items areparameters default values provided in the job. While executing the job , if the user wants to change this default values to the user defined entries, we have to make the following changes in the script.

/proj/sap/SBOP_INFO_PLAT_SVCS_40_LNX64/dataservices/bin/AL_RWJobLauncher /proj/sap/SBOP_INFO_PLAT_SVCS_40_LNX64/dataservices/log/DEV_JS_1/"

-w "inet:acas183.fmq.abcd.com:3500" " -PLocaleUTF8 -R\"REPO_NAME.txt\"  -G"1142378d_784a_45cd_94d7_4a8411a9441b"

-r1000 -T14 -LocaleGV -GV\"\$Character_123=$1;\$Integer_One=$2;\"  -GV\"DMuMDEn;\"    -CtBatch -Cmacas183.fmq.abcd.com -CaAdministrator -Cjacas183.fmq.abcd.com -Cp3500 "

Where $1 and $2 are the parameters passed by the user replacing the default values.

Thus the job should execute in the following way

. ./Job_Name.sh $1 $2

Areas of difficulty.

The user entries should feed to the script as encrypted data. For this encryption, the value should be encrypted using AL_Encrypt utility.

That means if user need to pass an integer value 10 for a parameter variable say $Integer_One in the job, then he cannot use $Integer_One=10; instead he has to pass“MQ” which is the result of the utility AL_Encrypt

Al_Encrypt 10;

Result : - MQ

I have created a Script that can resolve the issue to a very good extend.

Logic of the Custom Script

Name of the Script: Unix_JOB_CALL.sh

Pre-Requisite:- Keep a parameter file (Keep entries line by line in the parameter order under a name “Job_Parm_File”)

(As we have different scripts for different jobs, we can keep different param files as well. Whenever there is a change invalue needed, user can simply go and modify the value without changing the order of parameter)

Sample Script Code

rm -f Unix_Param_Temp_File;

touch Unix_Param_Temp_File;

chmod 777 Unix_Param_Temp_File;

cat Job_Parm_File | while read a

  do

     AL_Encrypt $a >> Unix_Param_Temp_File

  done

JOB_PARAMS=`tr '\n' ' '<Unix_Param_Temp_File`

. ./Test_Unix.sh $JOB_PARAMS

rm Unix_Param_Temp_File;

Through this you can execute the job with user defined parameter values using unix.

Just to have fun with DS..

• If you want to stop staring at the trace log and wait for it to show "JOB COMPLETED SUCCESSFULLY".

        You can use an easy aid in DS..

         Go to Tools --> Options...
         Then break out Designer and click on General.

         Then click the box that says: Show Dialog when job is completed.

                                   
        Now whenever your job completes, you'll get a little dialog box popping up to let you know.

• One of the annoying defaults in Data Services is that all the names in Workflows or Dataflows are cut off after 17 characters.

            So to fix this go to Tools -> Options

            Then break out Designer and click on General.

            Where it says: Number of characters in workspace name. Change the number 17 to 100.
            Click OK when it asks you if you want to overwrite the job server parameters.

Njjoy...

Hi this blog is meant to help the ABAP programmers and Developers with all the status messages, while posting an IDOC in SAP.

Also I mentioned the error reason and possible solution.

Hope this is helpful to all.

Sequence of Inbound and Outbound statuses

Starting statuses may be: 01 (outbound), 50 (inbound), 42 (outbound test), 74 (inbound test)

Thanks,

Mayank Mehta

hi All,

i am facing an issue while Running the .net sample Projects with Visual studio 2010 (Windows 7 64 bit Operating System)

"Could not load file or assembly 'dotnet_emdq, Version=1.0.4496.42818, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format."

Thanks,

sravan g

Hi All,

Here is a query to find where SYSDATE is hard coded in all BODS ETL jobs.

SELECT DISTINCT parent_obj

   FROM di_repo.al_parent_child,

  di_repo.al_langtext

  WHERE parent_obj_key=parent_objid

  and parent_obj_type<>'DataFlow'

AND parent_obj NOT LIKE 'Copy_%'

AND regexp_count(upper(text_value),'(SYSDATE)')>0;

This will help in identifying the jobs having SYSDATE hard coded and can change the code to make it as TABLE driven.

Create a function to call this TABLE and then pass parameters to it get the value and that value can passed to a Global variable in the BODS Script.

As you may know GV can be reused in many places.

Hope this will be helpful.

Thanks,

Abdulrasheed.

This past week SAP provided this data migration webcast to ASUG.

Abstract provided by SAP:

In this session we learned about the Rapid Data Migration package, which is combining SAP’s Rapid Deployment Solutions (RDS) with SAP Data Services and SAP Information Steward software.   Ready and pre-built for SAP’s cloud solutions Customer on Demand and SuccessFactors Employee Central, it provides a standardized and successful approach to planning, designing, and implementing data migration projects.

Figure 1: Source: SAP

Figure 1 shows SAP cloud solutions.    SuccessFactors and Ariba were through acquisitions

Figure 2: Source: SAP

Data migration is a challenge for all companies, but there are differences with cloud.

Data migration projects can be budget busters and run over time and budget.

Move to cloud for low predictable cost of ownership.

Figure 3: Source: SAP

If move to the cloud, you need to perform data migration for Successfactors/Employee Central

SAP’s Data Migration solution includes Cloud for Customer and migration for SuccessFactors Employee Central.

It eases the process of moving into cloud applications – migration from on premise to cloud.  It includes pre-built content.  It corrects, validates to provide usable data.

RDS

Figure 4: Source: SAP

SAP’s desire to make implementation easier for customers.

It includes software from SAP, special content to accelerate project management as well as the specific pace and service offerings (fixed price, fixed scope) – SAP point solutions.

Content, in the case of data migration, includes pre-built validation for back end SAP and migration content for a wide range of business objects in back end and toolset for data enrichment.

Whiteboard

Figure 5: Source: SAP

Figure 5 shows a whiteboard to get to the SAP Cloud – SuccessFactors

You use Information Steward to standardize name

You use reporting to see how data migration is going and to report data quality

See more information at service.sap.com/rds-dm2cloud (SMP logon required)

Load using SAP methodology, and use Information Steward, Data Services, reporting via BI tools

Figure 6: Source: SAP

Successfactors changes from USA to United States

SF expects “California” rather than CA

Data Services is used for extract, transform and load – for data migration go from source to target

Figure 7: Source: SAP

Figure 7 shows the scope of moving from HCM to SF Employee Central

Figure 8: Source: SAP

Figure 8 shows the scope of Cloud for Customer migration objects.

Figure 9: Source: SAP

Figure 9 shows the solution is fixed price, fixed scope work by SAP or qualified partners.

Data migration starter service includes knowledge transfer, medium complexity and enables the in-house team

Level of skill with Data Services varies from customers to customer

With data migration to cloud has more options

If own Data Services you can download RDS solution and then engage SAP or partners

Other option – if not own Data Services and but want data migration – SAP cloud organization can provide migration in a hosted data services environment  - software is not on-premise

Related Links:

Meet ASUG TechEd Speaker Emil - Kellogg's Data How to Deliver One Version of the Truth

Data Migration Education Class: TZIM4M

• https://training.sap.com/us/en/course/tzim4m-data-migration-with-sap-data-services-classroom-096-us-en/

Data Migration on service marketplace

• http://service.sap.com/rds  – Data migration packages are under Enterprise Information Management

Recorded Demo of SAP rapid data migration to cloud solutions:

• http://service.sap.com/rds-dm2cloud

Blogs on data migration

• http://scn.sap.com/people/frank.densborn/blog/2012/05/25/a-better-way-to-migrate-your-sap-data-with-rapid-deploymentsolutions

Hello Guys,

Often we skip or ignore some of the minimal things which may make your jobs to be executed in a faster way. For the very reason, I had consolidated some key points by which we can make the BODS jobs in more efficient with optimal consumtion of resources. This discussion might me more helpful and efficient to the beginers in this area.

1. Increase monitor sample rate. ex..to 50K in prod environment.
2. Exclude virus scan on data integrator job logs.
3. While executing the job for first time or when changes occur with re-run. Select the option COLLECT STATISTICS FOR OPTIMIZATION (this is not selected by default).
4. While executing the job second time onwards. Use collected stats.(this is selected by default)
5. Degree of parallelism (DOP) option for your data flow to a value greater than one, the thread count per transform will increase. For example, a DOP of 5 allows five concurrent threads for a Query transform. To run objects within data flows in parallel, use the following Data Integrator features:
• Table partitioning
• File multithreading
• Degree of parallelism for data flows
6. Use the Run as a separate process option to split a data flow or use the Data Transfer transform to create two sub data flows to execute sequentially. Since each sub data flow is executed by a different Data Integrator al_engine process, the number of threads needed for each will be 50% less
7. If you are using the Degree of parallelism option in your data flow, reduce the number for this option in the data flow Properties window.
8. Design your data flow to run memory-consuming operations in separate sub data flows that each use a smaller amount of memory, and distribute the sub data flows over  different Job Servers to access memory on multiple machines.
9. Design your data flow to push down memory-consuming operations to the database.
10. Push-down memory-intensive operations to the database server so that less memory is used on the Job Server computer.
11. Use the power of the database server to execute SELECT operations (such as joins, Group By, and common functions such as decode and string functions). Often the database is optimized for these operations
12. You can also do a full push down from the source to the target, which means Data Integrator sends SQL INSERT INTO... SELECT statements to the target database.
13. Minimize the amount of data sent over the network. Fewer rows can be retrieved when the SQL statements include filters or aggregations.
14. Using the following Data Integrator features to improve throughput:
   a) Using caches for faster access to data
   b) Bulk loading to the target.
15. Always views the SQL that Data Integrator generates and adjust your design to maximize the SQL that is pushed down to improve performance.
16. Data Integrator does a full push-down operation to the source and target databases when the following conditions are met:
• All of the operations between the source table and target table can be pushed down.
• The source and target tables are from the same data store or they are in data stores that have a database link defined between them.
A full push-down operation is when all Data Integrator transform operations can be pushed own to the databases and the data streams directly from the source database to the target database. Data Integrator sends SQL INSERT INTO... SELECT statements to the target database
Where the SELECT retrieves data from the source.

17. Auto correct loading ensures that the same row is not duplicated in a target table, which is useful for data recovery operations. However, an auto correct load prevents a full push-down operation from the source to the target when the source and target are in different data stores.

18. For large loads where auto-correct is required, you can put a Data Transfer transform before the target to enable a full push down from the source to the target. Data Integrator generates an SQL MERGE INTO target statement that implements the Ignore columns with value and Ignore columns with null options if they are selected on the target.

19. The lookup and lookup_ext functions have cache options. Caching lookup sources improves performance because Data Integrator avoids the expensive task of creating a database query or full file scan on each row.

20. You can control the maximum number of parallel Data Integrator engine processes using the Job Server options (Tools > Options> Job Server > Environment). Note that if you have more than eight CPUs on your Job Server computer, you can increase Maximum number of engine processes to improve performance.

I wanted to learn about how the SAP EIM platform, specifically Data Services, integrated into Hadoop. Being a bit of a techie at heart and the fact that I'm not really into reading manuals I thought I'd gets hands on experience and decided to install my own virtual machine with both Hadoop and Data Services running on it. I know this sort of defeats the purpose of Hadoop with it's distributed file system and processing capabilities but it was the easiest way for me to learn.

I'm no Linux expert but with my basic knowledge and with some help from Google (other search engines are available) I decided to install the Intel distribution on a Linux virtual machine. Intel utilise the Intel Manager framework for managing and distributing Hadoop clusters and is relatively straight forward to get up and running. Once installed this provides a nice, easy to use web interface for installing the Hadoop components such as HDFS, Oozie, MapReduce, Hive etc.. These can of course all be installed manually but that takes time and using Intel Manager allowed me to provision my first Hadoop cluster (single node) relatively quickly.

A detailed explanation of the different Hadoop components can be found on the Apache Hadoop site - http://hadoop.apache.org/

Once Hadoop was up and running the next step was to install Data Services. I decided to go with Data Services 4.2 and this of course requires the BI Platform so I went with 4.0 SP7 as Data Services 4.2 doesn't yet support 4.1 of the BI platform. I went with the default installation and used the Sybase SQL Anywhere database, that is now bundled with the BI platform install, as the repository for both the CMS and the Data Services.

As per the technical manual, Data Services can connect to Apache Hadoop frameworks including HDFS and Hive sources and targets. Data Services must be installed on Linux in order to work with Hadoop. Relevant components of Hadoop include:

HDFS: Hadoop distributed file system. Stores data on nodes, providing very high aggregate bandwidth across the cluster.

Hive: A data warehouse infrastructure that allows SQL-like ad-hoc querying of data (in any format) stored in Hadoop.

Pig: A high-level data-flow language and execution framework for parallel computation that is built on top of Hadoop. Data Services uses Pig scripts to read from and write to HDFS including joins and push-down operations.

Data Services does not use ODBC to connect to Hive it has it's own adapter so you must 1st configure this in the management console. The technical manual has all the details and it is fairly straightforward to configure. Make sure you have all the relevant jar files listed in the classpath.

The next step in my learning with Hadoop and Data Services is to create a demo scenario. Rather than invent one I'm going to use an existing demo that can be downloaded from the SAP Developer Network and adapt it to work with Hadoop rather than a standard file system & database. I'm going to use the Text Data Processing Blueprints 4.2 Data Quality Management which takes 200 unstructured text files, passes them through Data Services Entity Extraction Transform and loads the results into a target database.

I'm going to put these files into HDFS and read them out using the Data Services HDFS file format, pass the data through the standard demo data flow and then load the data into Hive tables.

The demo comes with a BusinessObjects Universe and some Web Intelligence reports so time permitting I may port these over to read from Hive as well.

I'll hopefully create my 2nd blog once I've completed this with my finding and a recorded demo.

Step1:

In DATA SERVICES MANAGEMENT CONSOLE goto Batch Job Configuration Tab and click on Export Execution Command.

This will create a .bat file with the job name (Job_TBS.bat) in the following path:

D:\Program Files (x86)\SAP BusinessObjects\Data Services\Common\log\

Step2:

Use the below script to check whether the respective .bat file exist in the below path.

exec('cmd','dir "D:\\Program Files (x86)\SAP BusinessObjects\Data Services\Common\log\"*>\\D:\\file.txt');

Step3:

Create a new job (J_Scheduling) to trigger the job which needs to be executed (Job_TBS).

Use the below script to trigger the job.

exec('cmd','D:\\"Program Files (x86)"\"SAP BusinessObjects"\"Data Services"\Common\log\Job_TBS.bat');

Now the job J_Scheduling will trigger the job Job_TBSusing simple script

Remember when organisations were content with quarterly reporting? It enabled them to plan for the next three months or even the rest of the year. Seeing what happened within their business over the last three months was enough!

    We are in the “Information Now” era. Give ME the information as it is happening, at the point of sales, in the supply chain, on my mobile. Move over stale and out of date data, its all about "how fresh is your data?"

    Fresh green data isn’t always easy to deliver into the right hands at the right time. System diversity, complexity, mobility, infrastructure constraints, data volume explosions are all there lurking to trip up and stall the data along its journey.

    Businesses are living with an ever-shrinking data batch window and it can be like trying to put the same weekly shop into smaller and smaller bags week on week.

    SAP’s HANA platform is about helping to relieve customers of this paradigm of being shackled by data latency issues. The equation of hyper performance serving up piping hot data will lead businesses to innovation.

    But when customers live with a heterogeneous system landscape how can they get high quality disparate data available faster and at less cost?

    One way is to move less data, only take what you need,  lifting only the things  that have changed. This will greatly help the customer achieve this goal. Change Data Capture is a key strategy  to unlocking data latency.

    I am going to examine this area of using one of the tools in the Hana Platform, Data Services. How it  can be used to apply the most appropriate mechanism for delivering fresh, high quality data through the use of applying various change data capture techniques, it’s a question I get asked a lot.  There are misconceptions that Data Services is a batch driven tool, but it can and does operate in this era of “Information Now".

    These posts will be based on my experiences with variety of customer engagements over the years using Data Services and I am looking to promote a discussion, provide ideas, perhaps save you time (there may even be something worth cutting and pasting into an RFI).  The next post will describe the two main options of delivering data: Push or Pull.

One of the first things to take into account when figuring out the best approach to change data capture is identifying how the data going to be delivered to the Data Services engine. Do you have to go fetch the data or is it dropped in Data Services lap. From a technology point of view usually when data is pushed it is traveling via a message bus, API or Web Service and when pulling data Data Services connects to the source systems and pulls it out.                                                                       

PUSH METHOD

Typically, but not always if a customer is using the push method, Data Services will be running real-time jobs that listen out for messages to arrive along a queue. In this scenario customers “usually” are sending already identified changes (inserts/updates/deletes) along the message bus in XML packages. (One example of this could be real-time SAP idoc’s over ALE).

Most of the hard work is done here for the Data Services developer, the changes are pre-identified and delivered and all that needs doing is deciding on how to apply the delta load to the target and perhaps add a few data quality and transformations along the way! This method is the way a real-time data requirement can be fulfilled using Data Services and is something more customers are looking to do to achieve near real-time data integration.

      PULL METHOD

This is the more traditional method of acquiring data and one that every ETL developer is familiar with, Data Services instigates the extraction process from the source system. This is typically a scheduled job set to run at an interval. These intervals can vary from an evening extraction to a batch process that runs at regular intervals  such as the micro-batch in the latest version of Data Services. The Data Services Micro-batch will keep polling the source data until certain conditions are met, this delivers a lower level of time based granularity and simplifies the process of configuring this approach.  Micro-batching can be really useful when customers are looking to trickle feed the data to a target but may not be able to switch on CDC functionality in the source system, micro batching is platform independent where as CDC in Data Services is specific to the type of database being sourced.

When using a pull method the logic to identify changes is more often then not built using standard functions within Data Services such as table comparison or CDC features, however, other options include polling directories for change files, working with Sybase replication server.

So to recap, the reason why customers would consider using the push method with Data services when:

• They require data to be moved and processed in real-time. An example of this would be real-time address confirmation and cleansing from an ecommerce web portal through web service request to the Data Services engine.
• When they have a message queue or SOA architecture mechanism for data delivery.
• Need real time interactions with the data.
• Data is typically being processing from a single primary source (reaching out to other systems could introduce unwanted latency into the real-time process)

Pull methods are used where:

• Data latency is less of an issue and the data is processed at different intervals possibly via a schedule.
• The extracts are driven from Data Services through a batch process.
• High volume bulk data.
• Data is integrated from many sources providing complex cross system integration. 
• Change data identification needs to happen within Data Services

That said there is no technical limitation on the number of sources and targets that can be used within a single batch or real-time dataflow and this is more of a general rule of thumb.   Something to keep in mind is that Data Services primary focus is on data integration and not process integration between different applications within a business architecture. This is where a customer would adopt SAP Process Orchestration (PO).

In the next post I will start delving into Source versus Target based change data capture.

Purpose:

The purpose of this document is to up-grade SAP Business Objects Data Services from 11.7/3.X/4.X to SAP Business Objects Data Services 4.2

Overview:

Environment Details:

Operating system: Windows Server 2008 64 Bit

Database:Microsoft SQL Server 2008 R2

Web Application:Tomcat

SAP Business Objects Tools:SAP Business Objects Information Platform Services 4.1 SP2; SAP Business Objects Data Services 4.1 SP2

Migration Tools:Data Services Repository Manager

Repository version:BODI 11.7 or BODS 3.X or BODS 4.X

Installation & Configuration steps to upgrade SAP BODS 3.X/4.X to SAP BODS 4.2:

Pre Installation Check List:

• Backup of repositories, configuration files & Data Cleanse files:
  • Backup of Local, Profile & Central Repository
  • Backup of Following Configuration Files:
    • admin.xml
    • sapconnections.xml
    • as.xml
    • dsconfig.txt
  • Backup of Data Cleanse Files
• Create a Check list of Number of Batch Jobs/Real Time Jobs, Number of Access Server, Number of Job Server & Configuration Path (Datastore) of Source & Target Systems
• Create a Check list of Jobs Scheduling in SAP Business Objects Data Services Management Console
• Create a Check list of Groups & Users Available in SAP Business Objects Data Services Management Console

Post Installation Check List:

• Installation of EIM(Enterprise Information Management) or IPS(Information Platform Services) Package in New Landscape
• Installation of SAP BODS 4.2
• Installation & Configuration of Address Cleansing Package 4.2
• Best Practices says use the same Data base that you have use in SAP BODS 4.1 make a cloned copy and Use in SAP BODS 4.2 environment
• Up-gradation of Local /Central repository Using Data Services Repository Manager
• Configuration of Repository in EIM or IPS (Package Name) Central management Console.
• Configuration of SAP RFC in Data Services Management Console 4.2
• Configuration of Adapters in Data Services Management Console 4.2
• Apply Rights & Security according to your requirement in BIM or IPS (Package Name) Central management Console
• Now Login in SAP BODS Designer 4.2 and configure Jobs (Real Time/Batch Jobs)
• Configuration of Datastore (SAPR3,SAP BW, RDBMS), Substitution Parameters, Job Server, Access Server, Format Data Type (Flat Files, CSV)
• Now validate the Jobs at Top Level and if error comes fix those errors
• Now Execute the Jobs using Scheduler or other BI Tools

SAP Business Objects Information Platform Services 4.1 SP2 Installation & Configuration Documents:

Prerequisite:

SAP BusinessObjects Business Intelligence (BI) 4.1 SP2 or its higher compatible patches*

OR

SAP BusinessObjects Information Platform Services (IPS) 4.1 SP2 or its higher compatible patches*

Up gradation & Configuration Documents:

When the prerequisite system check completes successfully, click Next

On the "SAP BusinessObjects Information platform services 4.1 SP2 setup" page, click Next.

Accept the License Agreement, and click Next.

Type a password for the CMS Administrator account, and click Next

To start the installation, click Next.

Installation is in progress

Finish

Up gradation & Configuration Documents of Data Services:

When the prerequisite system check completes successfully, click Next

On the "SAP BusinessObjects Data Services 4.2 SP1 setup" page, click Next.

Accept the License Agreement, and click Next.

Click Next to accept the default path to the folder where the program will be installed.

  Click Next to accept the default language (English).

Type a password for the CMS Administrator account, and click Next.

Click Yes to restart the BOIPS services

To start the installation, click Next.

Installation is in progress

Finish

SAP Business Objects Data Services 4.0 Configuration Documents:

Before upgrade of local repository below are the error in SAP Business Objects Data Services Designer

So before login upgrade the repository using SAP Business Objects Repository manager.Below are the steps for upgrade of  local repository details:

Below are the steps for upgrade of local repository & upgrade of secure central repository

Upgrade of Local Repository:

Upgrade of Secure Central Repository:

After Installation & Up-gradation of SAP business Objects BOE/IPS Server, SAP Business Objects Data Services 4.2SP1 below are the status that i upgraded in VM Ware

Login to Central Management Console -> Go to Data Services

Then Click on Data Services and update the repository details of SAP Business Objects Data Services repository.

Now try to login in SAP Business Objects Data Services Designer

Click on Logon

Click on “ok”

Now login in SAP Business Objects Data Services management console

Click on "Administrator" tab

New features are added in “SAP Business Objects Data Services management console 4.2":

Object Promotion:

• Import Configuration
• Export Configuration

Export Configuration using two way:

• FTP
• Shared Directory

Substitution Parameter : Now we can change the "Substitution Parameter" settings through SAP Business Objects Data Services Mananegment console also.

New Features added in Adapter like "Hive Adapter & VCF Adapter"

Changes in "Query and Validation" transform in SAP Business Objects Data Services Designer 4.2

Changes in "Architecture" of SAP Business Objects Data Services please refer below upgrade guide for more details.

New Features added like "REST web services": Representational State Transfer (REST or RESTful) web service is a design pattern for the World Wide Web. Data Services now allows you to call the REST server and then browse through and use the data the server returns

Relevant Systems

These enhancements were successfully implemented in the following systems:

• SAP Business Objects Information Platform Services 4.1 SP2/SAP Business Objects Enterprise Information Management 4.1 SP2
• SAP Business Objects Data Services 4.1 SP2

This document is relevant for:

• SAP Business Objects Data Services Administrator

This blog does not cover:

• SAP Business Objects Data Quality Management Up-Gradation

Reference Material:

Upgrade Guide:

https://help.sap.com/businessobject/product_guides/sbods42/en/ds_42_upgrade_en.pdf

Installation Guide Windows:

https://help.sap.com/businessobject/product_guides/sbods42/en/ds_42_install_win_en.pdf

Installation Guide UNIX:

https://help.sap.com/businessobject/product_guides/sbods42/en/ds_42_install_unix_en.pdf

SAP Data Services 4.2:

https://help.sap.com/bods

SNotes:

1530081 - Troubleshooting tips for SAP Data Services 4.x

https://service.sap.com/sap/support/notes/1530081

1723342 - Data Services 4.1 Upgrade Scenarios

https://service.sap.com/sap/support/notes/1723342

1725915 - Release Notes for SAP Data Services 4.1 (14.1.0) Ramp-Up

https://service.sap.com/sap/support/notes/1725915

1740516 - SAP Data Services 4.x and SAP Information Steward 4.x compatibility with SAP Business Intelligence platform and Information Version

https://service.sap.com/sap/support/notes/1740516

1900051 - SAP Data Services 4.x and SAP Information Steward 4.x compatibility with SAP BusinessObjects Business Intelligence platform and Information platform services for inactive releases.

https://service.sap.com/sap/support/notes/1900051

Hi All,

I am facing issue in connecting BODS designer from my local system to Client system,that is BODS installed on remote server.

The local system and client system are in same LAN connection. I am using the same credentials that are using on server side like system ID and port number of server machine.

By using these credentials we are facing connectivity issue as follows.

But by using these credentials, the local machine user is able to connect to BODS on server location by using same IP address mentioned in screenshot.

In same IP address CMC and Information steward is working fine by using web URL from remote machines, only Designer is showing problem in connectivity to server side system.

Kindly help us to resolve this issue with your valuable comments.

Where I work the SAP Utilities solution is the heart of our SAP system.  Therefore when it was decided to merge two more utility companies into our existing company, I have to thank our ASUG SAP point of contact Ina Felsheim  who over 2 years ago helped arrange a data migration ASUG webcast for Utilities.

The great Ginger Gatling  provided the webcast.  Ginger has since moved on to cloud integration, but I am grateful to both Ina and Ginger for this webcast.  First I am not an expert in the SAP Utilities solution - it covers the gamut of device management, customer service, billing, FI-CA, and so much more.

SAP provides a rapid deployment solution for utilities using Data Services, and included in this migration solution are templates to use.   You don't necessarily have to use the Data Services tool to even use the templates provided.  The spreadsheet templates helped provide a framework of the data mapping needed/required to fit into the SAP structures.

It also helped to have a SAP Experts subscription, as I listed to a recorded session given by SAP's Paul Medaille  - this session helped me find and locate the needed templates.

For Rapid Deployment Solutions, SAP recommends you obtain consulting to help with the solution.  We decided not to do this and do it ourselves.  We installed Data Services in-house, and then used the RDS templates to migrate the SAP Utilities objects such as PARTNER, ACCOUNT, DEVICE, etc.

In my mind, here are the basic steps of the migration:

Step 1: Download the SAP Mapping Templates

This helps provide the framework of the legacy data you have and what it should map to in SAP.  This helped me a great deal as I was not an expert in this SAP area.

Step 2: Get the Data Services RDS Solution for Utilities Up and Running

SAP provides RDS installation instructions.

Since I didn't have a background in Data Services, I also spent time in a 2 day SAP data migration class - I wrote about it here: Do you still need a live SAP Education Course?

I was very impressed by a co-worker who didn't attend any training and immediately understood how to use Data Services.

Step 3: Start Populating the Spreadsheets from Legacy to Feed to Data Services

This was the best part of the project, because you populate the spreadsheets by object and this services as a data source to Data Services.  Data Services then generates a file that can be read by SAP's migration tool EMIGALL.

Step 4: Use EMIGALL to Load SAP

Another co-worker spent time getting familiar with EMIGALL and it is a very handy tool to load data into SAP.  But to me, the Data Services solution did the heavy lifting for us non-SAP Utilities experts, to get the required, right data in a needed format.

I remember another ASUG customer presentation (Kellogg's) on data migration - the goal for data migration should be 0 defects.  Getting close...

If you want more details, links, information, please reply below - and I will try to find time to write more about this.