Wednesday, July 29, 2015

SAP Upgrades - How to estimate the time needed for a production upgrade


One of the biggest questions with an upgrade is to see if it can be completed over a cutover weekend. Are there enough resources available to complete the task within the needed time frame? Is there a way to tell beforehand or will additional resources need to be brought in?

For many clients answering these questions can be a challenge. Many do not have installations that match from one system to another (i.e. QAS doesn't have the same setup as PRD). Estimating the time needed to upgrade a production system needs to be based on the time it took to upgrade DEV and specifically QAS. Other clients have shared resources in one environment but not another.

In these cases, what you can do complete the upgrades in the non-production systems and create an estimate. Then, as you are stepping through the pre-processing phases of the production upgrade, revise those estimates.

Here is a quick example of what can be done.




 The screenshot shows the time it took to complete an upgrade in both the DEV and the QAS systems. What I've done here, is to pull the time from the report generated at the end of each of the upgrades. As you can see, the EQ1 system is a bit faster than the ED1 system. 

I'm not including the hardware here (at least not yet) because I want to make a quick point. Estimates for time it will take for the production upgrade based on the previous upgrades are just going to be general ball-park estimates. Those estimates are not going to be realistic until they are revised based on the time it takes for the few steps of the production upgrade.

Here is an update to the example above.
 Above, I've created a base estimate on the amount of time it should take for each step in the production upgrade. I know that production has more memory (24 GB versus 12 GB) and more CPU (8 cores versus the 4 in Q). So I've put in a rough 20% improvement in time for each phase.

I also updated the other two columns as a sanity check against my original estimate. Once I start feeding data in from the active upgrade, I'll know which estimates are closer (part of me always hopes that I'm closer to the right than the left!).



 Now comes the fun part. We've started the upgrade and completed the first three phases (normally in an upgrade you can complete these steps in a day or two).  I pull the time required from the first phases from the sapupconsole log file. It's messy, but I can calculate the time for each phase by extracting the time for each step in a phase and then adding them all up (and who said Basis wasn't any fun?).


















Based on what I calculate for the first three phases, I update the spreadsheet as shown below (adding a calculation to show the percentage difference between the 'real' time and the time it took to complete the same phase in EQ1). Then I check to see how far off I am from my original estimate (of 20%). As you can see, I'm off on the 20% estimate but pretty close to the 30% column (the number at the bottom of the column is an average).









What we now have is a more accurate view of the time it will take for each phase in the upgrade. The overall upgrade plan can be updated to reflect these times.

Thursday, July 23, 2015

SAP Upgrades - Key Players

When I left my last project, I sent out notes to my colleagues who were the most helpful. That note mentioned that if I had to put together a dream team for my next project, that I'd want them to be on that list. In the back of my mind, I started thinking of why specific people were so valuable.

So here is my list of who should go on a dream team for SAP upgrades:

I. The Team

1. Project Manager
2. ABAP Programmer (depending on what is being upgraded)
3. Database Administrator
4. Operating (Windows/Unix) Administrator
5. Storage/SAN Administrator
6. Basis Administrator (I almost forgot that one!)
7. SAP Architect (if different from above)
8. Company Sponsor


II. Explanation

Project Manager
The best projects I've worked on included project managers who constantly worked to keep the lines of communication open. When there was a delay in the project or a major stumbling block (failed updates, corrupted upgrade status files, etc.), they took care of the communication while the technical team worked the issue. When outside resources needed to be brought in, they also took the lead on providing background information and ensuring access.

ABAP Programmer
A good ABAP'er is needed during the SPDD/SPAU update process. It makes the upgrade smoother and the validation process shorter and more reliable. For ECC upgrades, it is especially important to have a programmer review needed changes and stage them for the next system in the landscape.

Database Administrator
This is a key player on the technical side. They need to be able to answer these questions:
- Does the DB have enough space/capacity to handle the upgrade?
- Has the DB been backed up!
- Is the DB out of log archive mode (going into the downtime)
- What is needed for any possible upgrade to a new database version (if that is included)?

Unix/Windows Administrator
As with the DB admins, these technical players need to know if the upgrade on an OS level is possible and answer these questions:
- Has the OS been setup according the the recommendations from SAP (you'd be surprised how often this is not true)
- For any HA environments, has the system been tested prior to the install (if that is possible)?

Storage/SAN Administrator
The storage admin needs to be able to:
- Ensure there is enough space (duh).
- Be able to resolve an I/O issues
- Make sure the system is being backed up at the OS level.

Basis Administrator
This is the key technical role during the upgrade. The Basis Admin prepares for and completes the upgrade on the technical side. They also need to have experience with previous upgrades, be able to work through routine upgrade issues, and know when to call for outside help when needed.

SAP Architect
For larger projects that involve external systems and/or High Availability setups, having an architect can be crucial. Being able to encompass all external systems during testing and ensuring that the environments all match can be critical during an upgrade project. An architect should ensure that the infrastructure

Company Sponsor
This person is critical because you need someone from the executive level to highlight the importance of an upgrade. When you don't have a key player that supports the application and the upgrade, you may face an uphill battle of getting resources assigned to the project and having those resources complete their tasks within the required time frame.

Friday, July 17, 2015

Taking a break and playing with Hadoop

Background - Why I'm taking a break
I've been working on SAP projects for over twelve years. This year, I decided to take a slight break to catch up on some of my skills that may be lacking.

At heart I'm a Unix/Linux fan. With a cheap price point and the ability to install on nearly any platform means I can work and learn at home and at my leisure. My latest project is to get Hadoop running locally on a Debian system. The steps below are those I recorded as I installed a simple system (these are really just my crib sheets for the next time I want to install Hadoop).

As a side note, this is how I create most of my documentation. Whenever I install, upgrade or migrate a SAP system, I capture everything! I use Snag-it to grab every screen shot that appears as I go from step to step. Even if I don't capture all of the screenshots in a document, I still have a running record of what was completed which Snag-it happily sorts into daily folders. There have been a few times when I could not recall what was completed (I keep paper notes as well) but I was able to pull up a screenshot that showed the exact steps.

I. System Preparation
First I installed a new copy of Debian (barebones style with no KDE/Gnome). Then I started prepping and installing the system by following the steps on this website (a big 'thank you' to Wei Wang):
http://www.drweiwang.com/install-hadoop-2-2-0-debian/

For Hadoop, we need to prepare the system with the following tasks (the Hadoop site has a great list here Single Cluster Install):
  1. Install Java (Hadoop requires at least Java 1.6)
  2. Create users and groups
  3. Setup SSH connectivity 

The first thing I did was to install Java as shown here:
sudo apt-get install openjdk-7-jre


Then I ran a quick check to make sure the right version of java is the default with this command:
java -version


Still following the steps on Dr. Wei's site, create users and groups as shown below:
addgroup hadoop
adduser --ingroup hadoop hadoopuser
adduser hadoopuser sudo





The system needs to be able to connect to itself over SSH. Generate the RSA keys as shown below to allow for local SSH connections to be completed without asking for a username and password.
ssh-keygen -t rsa -P ''


Update the authorized_keys file for user hadoopuser as shown here:
cd ~
cd .ssh
cat id_rsa.pub > authorized_keys
To test --> ssh localhost (you should not be asked for a password)



II. Download, install, and configure Hadoop
In this phase, we just complete the following:
  1. Download and extract the Hadoop binaries
  2. Rename Hadoop directory and set the ownership of the extracted files
  3. Update the environment variables for the hadoopuser account
For the download (I'm still following the steps on Dr. Wei's site), I changed to the /usr/local directory and then completed the download with the command:
wget http://apache.cs.utah.edu/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz



Then I extracted the sources files with the command:
tar -xvf hadoop-2.7.1.tar.gz

Next, rename the directory to just 'hadoop' with the command:
mv hadoop-2.7.1 hadoop


As detailed in Dr. Wei's steps, change the ownership of the newly renamed directory with this command:
chown -R hadoopuser:hadoop hadoop
This sets the the ownership to the hadoopuser user account and the hadoop group for all files under the hadoop subdirectory.


Next, we need to setup the environment variables for the hadoopuser account so the system knows where the hadoop resources are located (such as the binary and configuration file locations).  These variables are applied when the hadoopuser account logs into the system and executes the .bashrc file (as part of the login routine).

Update the environment variables with these steps:
cd ~
vi .bashrc



Add these lines to the bottom of the .bashrc file:
# For Hadoop
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386/jre/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_HOME=$HADOOP_INSTALL
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export HADOOP_YARN_HOME=$HADOOP_INSTALL
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop





Next comes some basic updates to the Hadoop configuration files. Here, the JAVA_HOME variable is updated in the hadoop-env.sh script.
cd /usr/local/hadoop/etc/hadoop
vi hadoop-env.sh

"

Update JAVA_HOME to:
JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386/jre/


To confirm that everything is working okay,  a quick test is run to see if Hadoop starts up.
cd /usr/local/hadoop/bin
./hadoop version



Following Dr. Wei's site, here are more configuration files that need to be updated:
cd /usr/local/hadoop/etc/hadoop
vi core-site.xml


Add these lines inbetween the entries:

   fs.default.name
   hdfs://localhost:9000



Update yarn-site.xml with these lines:


   yarn.nodemanager.aux-services
   mapreduce_shuffle


   yarn.nodemanager.aux-services.mapreduce.shuffle.class
   org.apache.hadoop.mapred.ShuffleHandler





Update the mapred-site.xml file with these lines:

   mapreduce.framework.name
   yarn






The Hadoop file system itself needs to be setup and initialized. First (following Wei's steps) create directories to hold HDFS data using these steps:
(as user hadoopuser)
cd ~
mkdir -p mydata/hdfs/namenode
mkdir -p mydata/hdfs/datanode


Then update the hdfs configuration file:
cd /usr/local/hadoop/etc/hadoop
vi hdfs-site.xml



Add these lines to the file:

   dfs.replication
   1


   dfs.namenode.name.dir
   file:/home/hadoopuser/mydata/hdfs/namenode


   dfs.datanode.data.dir
   file:/home/hadoopuser/mydata/hdfs/datanode



Format the namenode using the hdfs command:
cd /usr/local/hadoop/bin
./hdfs namenode -format




III. Start Hadoop and run some tests!

At this point, everything has been configured so we just need to start Hadoop and run some tests to validate the system.

Start hadoop with these steps:
cd /usr/local/hadoop/sbin
./start-all.sh



Check the processes with command: ps -ax
As you can see below, the namenode, the datanode, and the secondarynamenode are running as needed.


A quick visit to port 50070 on the system shows the datanode is running and HDFS is available as needed.


Finally, run a test using the mapreduce example:

cd /usr/local/hadoop/bin
./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 2 5



Check out the progress on port 8088


Confirm that everything complete without any issues.


So there it is. A full running Hadoop system on an old half abandoned system with half a gig of memory. Not bad!

Next I just need to setup a distributed environment, use a faster box, and finally learn the power of this sophisticated software.