SAP XI component monitoring (XΙ 3.0)

After a short period of assisting in the troubleshooting of a problematic SAP XI 3.0 installation, I propose a set of monitoring transactions, that can (maybe must) be used in a daily basis (from XI/Basis people) to make sure the XI server is running well. First, some low-level pre-requisites:

1. If you are stupid enough to setup the SAP/XI server on Windows (particularly 32-bit) OS (and/or be responsible for it), make sure that the kernel has enough resources to handle the zillion connections required for XI operation. You will NEED to keep the Memory / Free System Page Table Entries (PTEs) at least 20.000. You can monitor this with "perfmon.msc". If this number drops well below 20k, you will experience sever problems: dropped connections, internal time-outs and plenty of HTTP errors (401, 503, etc). Anyway, check the boot.ini switches /USERVA, /3GB, /PAE and set them following Microsoft rules.
2. Bare in mind that XI requires plenty of power and resources. It starts several virtual machine instances (both ABAP and JAVA/J2EE) and handles heavy network load. These need plenty of memory and a good OS kernel. When setting up an XI server, these things should be taken into account.

The XI Directory and XI Repository have nothing to do with XI monitoring. Everything needs to be done from the R/3 system itself, and/or the XI Runtime Workbench (RWB). First, the XI-related SAP transactions, which help you find out if everything is OK:

• XI Administration: SXMB_ADM. Manages every aspect of XI operation from the R/3 end, including queues, archiving and deletion, as well as various system-wide definitions.
• ICM Monitor: SMICM. Supreme overlord for Java maintenance from the R/3 end. Permits J2EE restarts (Check "Administration" menu) and offers useful Java monitor stats.
• Queue monitors: SMQ2, SMQ1, SMQ3. If any message is stuck in the way in or out of the XI switch, you'll find it here. Watch particularly for SMQ2. If you are unlucky to find out that a certain queue is stuck (the top message failed to get processed), select the message and press SHIFT+F6 (Save LUW). This will send it at the end of another queue, permitting processing of the next messages. Note: XBTI* are queues for incoming messages, XBTO* are queues for outgoing messages.
• Message monitor: SXMB_MONI. Find recent messages with errors (inbound or outbound) and generally anything that goes through XI. Perhaps the most useful transaction, but with rather limited search criteria options :-(
• Business Process Engine monitors: SXMB_MONI_BPE. A useful list of monitoring tasks for the status of the XI Processing Engine.
• BPE Errors: SWF_XI_SWI2_DIAG. A sub-section of BPE monitors for latest errors.
• Outgoing RFC monitor: SM58. Find out if another R/3 system has problems processing the messages that XI sent.
• Local RFC Monitor: SWF_XI_SWU2. Monitor the XI ABAP processing status.
• Hardcore message monitoring/retrieval: SE16. For hardcore problems, you can always resort to the tables, that XI uses to store (persist) transferred messages. Look into tables: SXMSPMAST (master table), SXMSPERROR/SXMSPERRO2 (error message info), SXMSPVERS, and some other. NOTE: XI_AF_* tables, that exist in the SAP XI DB (part of the J2EE adapter framework), are NOT managed by R/3 and do NOT exist in the ABAP workbench.
• IDoc monitoring: IDX5. Useful for finding the status of certain R/3 objects (materials, sales/purchase orders, etc) that seem to be lost in the XI way. You need the IDoc number, the Message GUID, or a combination of time-date and IDoc type. Extremely useful when checking with other modules/systems and SAP users.
• TCP/IP listeners: SMMS. See where the XI engine listens for incoming client connections in the TCP/IP level. Quite useful in conjunction with netstat, telnet, wireshark/ethereal and other networking tools.

Another set of important monitors is within the Runtime Workbench (RWB), as a part of the web-based XI components. With RWB, you can:

• Monitor Communication Channel status: Check channels for hidden errors (e.g. "why my file does not get parsed from XI?"); start/stop channel service (common solution for old XI systems); monitor produced/consumed messages.
  Access from: Component Monitoring / Adapter Engine / Communication Channle Monitoring.
  URL: http://<server_ip>:<adm_port>/rwb/index.jsp
• Monitor Messages: Just like SXMB_MONI, but this is more reliable, in some cases.
  Access from: Message Monitoring.
• Check user locks: Remove stale locks from missed connections, etc. in Repository/Directory.
  Access From: XI Tools / Administration / Repository / Lock Overview
  URL: http://<server_ip
  >:<adm_port>/rep/support/public/LockAdminService

All XI messages follow the SOAP/XML scheme. When a specific problem is found with a message, it is always good practice to look into its Trace section, apart from the Error section. Most likely, the trace contains a Java stack trace, from which, the top entry is of interest.

My first webapp, Walkthrough pt2: Server configuration

In order to prepare a system for the db-enabled webapp, follow the steps below:

1. Download+install Java (preferrably Java6 SDK). Setup $JAVA_HOME environmental variable accordingly, so that $JAVA_HOME/bin/java is the JVM executable. For linux, package managers now offer sun-java-* packages, you'd better AVOID the gcj stuff, for maximum compatibility.
   
2. Download+install MySQL. Again, linux distros have packages in default repos. Edit the MySQL configuration file (/etc/mysql/my.cnf or whatever), so that, under "[mysqld]", you COMMENT the "skip-networking" option and give a good bind address (127.0.0.1 is safest). Part of the file will look like
   [mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/English
bind-address = 127.0.0.1
# skip-networking
The example listens only on the local loopback interface, change for use with remote tomcat. In general, test access to your mysql daemon with "telnet localhost 3306", or such.
   
3. Download+install Tomcat-5.5, preferrably the zip package. Linux distros offer tomcat packages. Consider $CATALINA_HOME to be the installation path, so that $CATALINA_HOME/bin/startup.sh starts the server. Tomcat requires special administration:
1. Download the MySQL JDBC connector, and place the JAR file in $CATALINA_HOME/common/lib, for all webapps to see. Keep in mind that the MySQL driver class is com.mysql.jdbc.Driver.
2. See to that the Security Manager lets the driver create a socket to the listening server port, by adding a policy entry to $CATALINA_HOME/common/conf/catalina.policy:
   grant codeBase "jar:file:/path/to/tomcat5.5/common/lib/mysql-connector-java-5.1.5.jar!/-" {
//permission java.net.SocketPermission "localhost:", "connect";
//permission java.net.SocketPermission "127.0.0.1:3306", "connect";
//permission java.net.SocketPermission "127.0.0.1:3306", "resolve";
permission java.security.AllPermission ;
};
   
3. 
   
4. For development, NetBeans 6.1 is good, Eclipse offers some not-so-good plugins, and Vi is always a safe choice.

My first webapp, Walkthrough pt1: Requirements

This is the first entry in a series of post that guide through the creation of a tomcat (tomcat-5.5) web application with the following features:

1. Database connectivity (mysql-5)
2. User authentication (a simple one, not JAAS stuff)
3. User-created database content
   
4. Universal foreign language support in forms, parameters and files (utf-8 encoding)
5. File creation/manipulation
   

I consider a java-6 based system, and will report which things differ with standard java-1.4 cases (usually the encoding stuff). There will be no reference to the development tools, since this walkthrough aims at full understanding of every potential change to the configuration of the involved software subsystems. I would suggest any linux flavour for this. My configuration:

• Linux Mint 4.0 (ubuntu-7.10 based)
  
• Sun Java 6u6 (from apt-get)
• Tomcat 5.5 (from apt-get)
• Mysql 5.1 (from apt-get)

The Walkthrough pt2 will hold important configuration parameters of the installed software

Rollback with svn

Trickier than it may seem, but simpler than expected. This is how you revert/rollback to an earlier svn version and cancel/undo all the stupid things you have done. Example:
Assume: $STUPID_VERSION is the version that messed things up, $LOCALMODIFIEDDIR the place where things change (must be svn-controlled of course, but can be a subdir of the project, if this is desired)

$ cd $LOCALMODIFIEDDIR
$ svn merge -c -$STUPID_VERSION .

Note the dot at the end '.'; Found here, at this good guy's blog.
.. oh, and NEVER NEVER NEVER copy entire svn-controlled directories!!! the ".svn" hidden dirs are chaotic evil.

Now, it's quite probable that a future commit will result in failure because of "file already exists"... Therefore, make a local copy of the affected $LOCALMODIFIEDDIR somewhere without svn (let it be $BACKUPDIR), DELETE the ".svn" inside, and:

$ svn delete --force $LOCALMODIFIEDDIR
$ svn commit -m "deleted the dir"
$ cp -r $BACKUPDIR $NEWDDIR
$ svn add $NEWDIR
$ svn commit -m "stupid error fixed "

So simple... :-(

Windows codecs and codec packs

It has become more and more difficult to find windows codec packs that do not include adware/spyware/bloatware. It is perhaps best to get things from their source, it is not too difficult. Most movies that cannot be played by Windows Media Player (WMP) may include one or more of the following media types:

• Some MPEG-4 encoded video stream (DivX-style), which is almost always playable by xvid. Windows installable packages can be found there. Also FFDshow has become quite popular for that. [99.9% of movies need this]
• An MP3 encoded stream. WMP can deal with that, you should have no problem. [80% of movies need this]
• An AC-3 encoded stream. The AC3Filter takes care of these high-quality audio streams. Beware of the volume level!!! Usually one should give significantly more gain from the AC3 control panel entry in order to achieve good volume levels. [>15% of movies need this]
• An OGG-Vorbis audio stream. The Ogg-vorbis decoder can be found from Xiph.org here. [<2%>
• Other codecs are significantly less common (i think), like theora, or are of course supported by WMP inherently (all WMVs, MPEG, WMAs fall in this category).
• The Adobe Flash video files (FLV) can also be played by certain players, and there are also tools to save flash video streams (youtube, myspace) as FLV files.

It is usually important to support loading of subtitle files from WMP in an automated way. This is the job of VSFilter/DirectVobSub, which can be found from the guliverkli sourceforge project. This is a significant aspect, do not ignore this.

In order to obtain audio/video codec information for a file, it is usually sufficient to right-click on it and select "Properties". The last tab shows important information, like dimensions, codecs and data rates for the stream.

The good old codec packs (Nimo, KLite) become constantly more bloated and infected with adware, spyware, and other useless junk. I am going to check out the X-Codec-Pack, which seems to have important stuff and work with minimum intrusion. In addition - and in contrast with most cases - prefer installable downloads (*.exe, *.msi files), which do write useful registry information automatically.

Install linux without burning CDs/DVDs

The fundamental part of the process of installing linux without burning CDROMs or DVDs ALL the freaking time, is booting your machine into linux from within windows, with GRUB for NT.
This new loader allows booting a linux image using the NT boot loader. Now, selecting the right linux kernel and initrd image is something else. The UNetBootin project claims that it automates this process in an easy way...
My experiences on using these tools will follow....

My Embedded Project

Mini-Box Embedded Hardware
This site features many products suitable for developing an embedded (Linux?) system.

Linux Devices
This is actually a news site for embedded linux systems. Quite interesting implementations.

Slashdot: ZFS and NAS
A /. discussion about Network Access Storage (NAS) and cheaper implementations. This is only one part of the picture.

LAMP stuff

Tuning LAMP systems [IBM]

A comprehensive (but simple) article for optimizing PHP with Apache.

Graphics, Games and Drivers

THG's comparison of Nvidia and ATI drivers

2006, mid-July comparative test. Quite intuitive. Conclusions can be:

1. Nvidia drivers are more feature-full, slightly easier to install and provide more functionality (overclocking, SLI)
   
2. Tests showed no significant performance difference (ATI slightly better, Nvidia more consistent)
3. Gamins is not only for Win32
   

Database and Web CMS stuff

PHP, MySQL - Optimising MySQL queries - LXF Wiki

Optimizing MySQL queries, for PHP and not only. General information on SQL queries...

Apache Derby, ODF, DB2 - Portable Document Storage - IBM DeveloperWorks

A fine tutorial for making your own content management system for Open Document Format files, with example code for Apache Derby AS. [Needs IBM account]