Buy me an espresso

September 15, 2008

SharePoint Server Search toasted ... stuck on "crawling full"

Overview

It's the same old story. You wake up one morning and errors are flooding into the web server. This time it is your friend Mr. 10036. And he brought his SQL Server buddy 17310.

Event Type: ErrorEvent
Source: Office Server SearchEvent
Category: Gatherer
Event ID: 10036
Date: 9/15/2008
Time: 1:02:55 AM
User: N/A
Description: A database error occurred.
Source: Microsoft OLE DB Provider for SQL
ServerCode: 0 occurred 257 time(s)
Description: Unspecified error
Context: Application 'SharedServices'


Event Type: ErrorEvent
Source: MSSQLSERVER
Event Category: (2)
Event ID: 17310
Date: 9/15/2008
Time: 5:29:08 AM
User: N/A
Description:A user request from the session with SPID 58 generated a fatal exception. SQL Server is terminating this session. Contact Product Support Services with the dump produced in the log directory.
Data:0000: 9e 43 00 00 14 00 00 00 žC......0008: 08 00 00 00 4d 00 4f 00 ....M.O.0010: 53 00 53 00 44 00 45 00 S.S.D.E.0018: 56 00 00 00 00 00 00 00 V.......


The short version is that the Office SharePoint Search Service is malfunctioning. The content source always believies it is "crawling full" even after web server and database server reboots.


Side note, the User Profile store also utilizes the Search Service to do its updating/querying. It believes that it is constantly "importing" profiles ... which it is not.

Solution

Here are some possible solutions that may work:
  • Install SQL Server 2005 SP2 if it is not already installed
  • Reset all crawled content (if possible)

Ultimately, we had to recreate the SSP to fix the problem ... this is not as scary as it sounds:

  1. Go to Shared Serivices Administration
  2. Select New SSP
  3. On the New SSP page, you can select you current MySite web application IF you don't care about losing your user profile images.
  4. The rest of the information for the New SSP page should be self explainatory ... if it isn't seek help.
  5. After the new SSP is created you will need to reenter ALL values for ... everything. Don't worry your old SSP is still there and you can just pull it up in another browser tab and compare back and forth until you have it all filled in.
  6. Next change the associations of the new SSP to associate with the port 80 and other sites/apps you have
  7. Finally, change the new SSP to your default SSP
  8. Run the search and profile import to confirm your issues are gone.

June 16, 2008

SQL Server Migration: 2000 32bit TO 2005 64 bit (Part 1)

Overview
This is the first post in a 3 part series on how to migrate from SQL 2000 32bit TO SQL 2005 64bit.

The first part will be a migration from SQL 2000 32bit to SQL 2005 32bit.

Notes
  • Depending on what the Upgrade Adviser Tool tells you ... your steps may vary from those below.
  • We also have aliased our DB using the Network Client Utility. This is not necessarily best practice, but it allows for easy migration and DB name changes.

Steps
1. Backup your SharePoint farm completely
2. Run the MS SQL 2005 Upgrade Adviser Tool
3. Delete excess backup history in the (Master>backupset table) that goes over 10,000 rows. (speeds up migration)
4. Quiesce the Farm.
5. Stop the WWW publishing service and all SharePoint specific services.
6. Check to make sure all sessions are killed.
7. Backup the SQL 2000 DATABASES.
8. Add the accounts for the application pools to the main Administrators group on the SQL 2005 box.
9. Move all SharePoint related logins to the new SQL 2005 instance.
10. Restore the SQL 2000 backup databases to the new SQL 2005 box.
11. Modify queries that use column aliases prefixed by table aliases in the ORDER BY clause. (what the Adviser told us)
12. Run the following stored procedure: sp_fulltext_service ‘load_os_resources’, 1 (what the Advisor told us)
13. On the web server go to Start > Programs > Microsoft SQL Server > Network Client Utility
14. In the Alias tab change the IP from the old server to the new server
14. Rebuild the indexes on the 2005 databases
(what the Advisor told us).
15. Rebuild the statistics on the 2005 databases
(what the Advisor told us).
16. Reboot the web server.
17. Restart all services.
18. Unquiesce the Farm.

Part 2 of this series will be changing the compatibility of the DB from 80 to 90. Part 3 will be moving from SQL Server 2005 32 bit to 64bit.


April 18, 2008

Content Deployment ... Violation of PRIMARY KEY constraint ...

UPDATE
Hotfix 950279 is available via contacting MS Support. After some "wrangling" ... this worked for us.

Overview

If you have just ran an "all content" deployment successfully ... don't start celebrating yet. Sure, it may have taken over a week with a hotfix or two to get it working. Just keep in mind that you will most likely want to setup deployment on a schedule for "only new, changed" content. That process turns out to be just as fun.

Issue

The first attempt at running a "only new, changed" content deployment (after a successful "all content" deploy) ended quickly with the following error:

Violation of PRIMARY KEY constraint 'PK__#ExportObjects____2F364608'. Cannot insert duplicate key in object 'dbo.#ExportObjects'. The statement has been terminated. at ...

Solution for pre-SP1 installations

Hotfix 936867 will fix this issue IF YOU HAVE NOT INSTALLED (the SP1s).

No Solution Yet for SP1 ...

If you have installed the SP1s, there is a much more drawn out solution. If you have previously installed 936867 and you install SP1 after ... your error will most likely return. Microsoft claims in their description of the MOSS SP1 ("2007 Microsoft Office servers Service Pack 1") that 936867 is included in the SP ... and this is it one of the "Issues that the service pack fixes" ... um LIE!

Great discussion about this issue here on MS's SharePoint forum.

April 15, 2008

Reporting Services ... Customizing the RSViewerPage.aspx Page

Overview ... Monday morning QB
Upon installation of Reporting Services there is a ReportServer folder created under the LAYOUTS directory that contains the .aspx files necessary to display reports. The primary file used for the display or reports is RSViewerPage.aspx. This page contains a large web part called ReportViewerWebPart. This web part includes both the toolbar/actions menu of the report, the report itself, and the parameters pane ... all in that one webpart. Mistake #1 by MS.

Another fun fact I noticed is that these pages are not child pages of a master. They are just stand alone .aspx files. Mistake #2 by MS. Hints, these pages do not include the global placeholders/navigation of the default.master nor the neighboring application.master (also found in the LAYOUTS directory).

When you deal with MS support these are they types of answers you get when you bring up such issues ... basically, wait till the next version.

Goal
Since by default the RSViewPage.aspx does not contain the global SharePoint navigation ... let's take a look at how to include it in the report page.

Solution
1. Make a backup of
RSViewPage.aspx

2. Register the following at the top of the page:



3.
Inlcude the following references inside the HEAD tag:




4. Place the following OnLoad event inside the BODY tag:
onload="javascript:if (typeof(_spBodyOnLoadWrapper) != 'undefined') _spBodyOnLoadWrapper();

5. Add the folowing lines of code:
  • The ms-globalTitleArea TD class will contain the background image for the theme.
  • The Logo.gif image represents the site logo
  • Use id=onetIdTopNavBarContainer and class=ms-bannerContainer for the TopNavBar
  • The wssuc:TopNavBar is the SharePoint top navigation.
That should do it!

I ran into some other issues that were installation specific ... so don't expect this to work perfectly for everyone ... but it will get you at least 90% of the way there.

March 25, 2008

64 bit database migration (the easy way)

DB Installation: Win 2k3 R2 SP2, SQL Server 2005 32 bit
Goal: Win 2k3 R2 SP2, SQL Server 2005 64 bit

If you are planning on migrating from 32 to 64 bit for your SQL Server ... look no further. After reading though blogs, white papers, and books ... I decided to that there was an easier way to handle all this nonsense. In simple terms:
  1. Backup the databases on the 32 bit DB server (let's call it SharePointDBServer)
  2. Take the web server offline, stop all services, sites, app pools
  3. Take SharePointDBServer offline
  4. Bring up a different 64 bit DB server with the same name as the old one (SharePointDBServer)
  5. Restore the backups to the 64 bit DB server.
  6. Reboot the web server
  7. Restart any services on the web server that were stopped
That's it!

Keep in mind that if you have a DB server naming convention like SQL64SharePoint ... it probably won't work ... or if you cannot take the 32 bit DB offline ... that won't work either, etc, etc ...

If you have the means, this is ridiculously easier then trying this through STSADM ...

Things to remember:
  1. Pay attention to the Owner of the databases after they are created on the new 64 bit server ... this will most likely be domain/Administrator ... which may not be the Owner of the previous DBs.
  2. The services running on the new DB server will not be using the identities that were customized for the old DB server ... you will need to put in the correct user for some of those services.

March 12, 2008

Content Deployment ... 5323 and 4958

Overview

While attempting a simple content deployment the following errors where holding back the import phase ...

Event Type: Error

Event Source: Office SharePoint Server
Event Category: Content Deployment
Event ID: 5323
Date: 3/10/2008
Time: 10:22:57 PM
User: N/A
Computer: ServerName
Description:
Failed to transfer files to destination server for Content Deployment job 'Entire Site Collection'. Exception was: 'System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
--------------------------------------------------------------------------------------------------------------------------------------------------
Event Type: Error
Event Source: Office SharePoint Server
Event Category: Content Deployment
Event ID: 4958
Date: 3/10/2008
Time: 10:22:58 PM
User: N/A
Computer:
ServerName
Description:
Publishing: Content deployment job failed. Error: 'System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine

Solution

Turns out, this is related to "TCP Chimney" (read more here). There a registry value that needs to be changed here: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
in order for the content deployment to function.

Set the value of
EnableTCPChimney = 0.

OR Command line: Netsh int ip set chimney DISABLED

Then reboot the server.

Setting the following values to zero will also be include in a soon to be released hotfix ...

  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters - EnableRSS = 0
  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters - EnableTCPA = 0
  • February 15, 2008

    15 days later ... more 6398, 6482, 7076s

    Overview
    They are back! After 15 long days of clear event logs (that's very long for SharePoint) the errors have returned.

    While the "Attempted to read or write protected memory. This is often an indication that other memory is corrupt." specific errors are a memory ...

    The new kids on the block are "Not enough storage is available to process this command.".

    Solution
    Installed the .NET 2.0 SP1 (available through Windows Update) and the errors have been gone for 11 days now ... that doesn't mean they won't be back!

    January 22, 2008

    6389, 7076, and 6482 errors flooding in ...

    Overview
    Don't be surprised if you wake up one morning to a hot cup of 6482s with a side of 7076s and 6389s. They are related to threading and the SharePoint Timer Service.

    "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

    Up until recently, the only deterrent for these errors was to either 1) restart the Timer Server on a schedule, 2) reboot the server every other day, or 3) the 923028 hotfix. Of course, those are not solutions.

    Solution
    Six months later, Microsoft to the rescue! Microsoft always reminds me of the "drunken" (off red kryptonite) Superman in Superman 3. You know, when he had the 5 o'clock shadow and showed up like 2 hours late ... yeah.

    Microsoft just released a hotfix (946517) for these errors ... it worked for us ... good luck!

    NOTE: As of 2/4/08 ... you will have to contact MS Support directly to get the hotfix!

    December 21, 2007

    Installing Service Packs (WSS 3.0 SP1 and MOSS 2007 SP1)

    Microsoft offers extensive information on the requirements and steps for installing. Unfortunately, in MS style, the information is all over the place and the "high points" are buried in 3000 word articles.

    The Important Stuff:
    • You must install the WSS 3.0 SP1 BEFORE the MOSS 07 SP1
    • You must run the SPs on all web and application servers (if you have a seperate server for indexing, Excel, etc ... the SPs must also be ran on those).
    • The SPs do not need to be ran on the DB server.
    • You must use a configuration/installation account for the install of both Service Packs
      • Suggestion: Use GOD for this installation ... SA of the DB server and Administrator rights on the front end web server(s) ... then back out permissions after installation
    WSS 3.0 SP1 & MOSS 2007 SP1 Step-by-step (simplified):
    1. Backup the Farm
    2. Remove any orphaned objects using -databaserepair
    3. Quiesce the Farm
    4. Stop the WWW Publishing service
    5. Run the WSS 3.0 SP1 installer
    6. Agree to terms and click Continue
    7. Once the installer is finished the SharePoint Products and Technologies Wizard will appear ... CANCEL THE WIZARD
    8. RUN the MOSS 2007 SP1 installer
    9. Agree to terms and click Continue
    10. Once the installer is finished the SharePoint Products and Technologies Wizard will appear ... click Next
    11. A dialog box will come up saying "You must run setup to install new binary files for every server ..." ... click OK
    12. You will get a warning about services restarted ... click Yes
    13. The Wizard runs and hopefully finishes without any errors
    Microsoft Links:

    December 19, 2007

    Abosulte URLs in the Content Editor Web Part

    Overview
    For those of you that are unaware, the Content Editor Web Part (when using the Rich Text Editor) will change your image or link URLs to absolute paths (whether you like it or not). Obviously, this has a number of unfriendly consequences. If you are reading this post then you are suffering from one of those consequences ... I won't bore you with a list of the posssible issues this creates.

    Solutions

    Option 1 - Keep using the CEWP and manually remove the absolute paths on they images and links in the Content Editor Web Part.

    Option 2 - Use Telerik's RadEditor Web Part

    Option 3 - Use an event handler to remove the absolute path. Dan Attis' Blog

    Option 4 - Use the Publishing Control RichHTML fields. You will have to have the Publishing Feature activated on your site for this to be an option. More details here.

    December 18, 2007

    6641 and 6482 errors ...

    Overview
    For starters, you should not have any local Administrators running a service or application pool for SharePoint 2007. As tempting as it may be to resolve errors by simply adding them to the Administrators group, it is simply not necessary.

    Issue
    When you start to back out/restrict permissions after installation/configuration ... you will most likely start to see a number of errors popping up (6641 and 6482). They are permissions related.

    Event Type: Error
    Event Source: Office SharePoint Server
    Event Category: Office Server Shared Services
    Event ID: 6641

    Date: 12/15/2007
    Time: 2:00:00 AM
    User: N/A
    Computer: COMPUTERNAME
    Description:
    The SSP Timer Job User Profile Change Job was not run.

    Reason: Logon failure: the user has not been granted the requested logon type at this computer


    Solution
    For each of these steps I recommend restarting IIS and affected services ... or if you can (to eliminate any doubts) ... reboot the server.

    Step 1 - Check to make sure your service accounts have "Log on as a Service" permissions.
    • Go to Start > run > gpedit.msc ... to pull up group policies (shown right).

    • Computer Configuration > Windows Settings > Security Settings > Local Policies > User Rights Assignments

    • Double click "Log on as service" and check to see if your service accounts are listed under the "Local Security Setting" tab

    • If they are not there they will need to be added

    • Double click "Log on as a batch job" and check to see if your service accounts are listed under the "Local Security Setting" tab

    • If they are not there they will need to be added

    • If you have updated your policies you will at least need to run gpupdate/force from the command line.
    Step 2 - Add the service accounts to the Backup Operators local group.
    • The Backup Operators have a few more policy permissions that can tackle some of those pesky errors.
    Step 3 - Add the service accounts to the WSS_WPG group.
    • There are a number of posts out there that tell you to go into component services and manually add the service accounts to IIS WAMREG or SPSEARCH. DO NOT add them individually. SharePoint has already populated the WSS_WPG group to these DCOM applications.
    Worst case, you will need to add more permissions for the WSS_WPG group via component services (explained below):
    1. Start > Programs > Admin Tools > Component Services

    2. Computers > My Computer > DCOM Config > IIS WAMREG (and SPSearch)

    3. Right click > Properties > Security tab

    4. Under Launch and Activation Permissions click Edit

    5. Check the checkbox for Remote Launch and Remote Activation

    6. Click OK

    November 12, 2007

    Options for setting up a MOSS 2007 development environment ...

    There are a number of ways to setup your SharePoint 2007 development environment. Below I will highlight the primary options that should be considered. Each option can make sense in the right set of circumstances.

    Please keep in mind, I am not going into details about domain controllers, AD, Exchange, etc ... once you add those into the mix, there are easily 10 different ways to set it up ...

    Couple of no contest items:
    1. VMware performs better than Microsoft Virtual Server (at least for now)
    2. Installing and running SQL Server and IIS on the same networked machine is a security risk
    What each option should have installed:
    • Windows Server 2k3 SP2 +
    • SQL Server 2005 (should always be installed on its own machine/virtual)
    • .Net Framework 3.0
    • SharePoint 2007 (MOSS 07, WSS 3.0)
    • Visual Studio .Net 2005
    • SharePoint Designer

    OPTION 1 - DESKTOP VIRTUALIZATION

    Hardware: Desktop PC (with min 2GB RAM, running at least XP Pro SP2)
    Software: Virtual PC or Virtual Server

    When to consider: Virtual PC and Virtual Server are 100% free. If cost is a consideration, this is the cheapest solution. If your developers are competent enough in server administration and knowledge of how to lock down a box, the "up and running" time should be quick.

    Pros:
    - Cheap solution
    - Quick setup if you have competent developers
    - Allows developers to have a fully emulated environment at their disposal
    - Developers do not affect other as much (IIS restarts, errors, and debugging happen in isolation)
    Cons:
    - Developers are responsible for their own install ... ie, they can create a security risk with their virtual, they can screw it up to a point where they have to reinstall/reconfigure often ...
    - If you have a lot of developer turnover, it would be better to set it up using server virtualization
    - Desktop PCs need to be high performance machines


    OPTION 2 - SERVER VIRTUALIZATION

    Hardware:
    ESX, VMware Server
    Software: VMware Server requires a base OS ... ESX does not

    When to consider: If there is a strong separation between server administrators and developers in you organization, you have a lot of developer turnover, you can afford the biggest and best.

    Note: In larger companies, I usually see the server setup/administering/maintenance by a server administrator ... not developers.

    Pros:
    - Allows developers to have a fully emulated environment at their disposal
    - Developers do not (always) have complete control so they can't screw up installation or get the server out of wack with production
    - Developers do not affect other as much (IIS restarts, errors, and debugging happen in isolation)
    Cons:
    - Cost for the hardware
    - Setup time can be longer than other options
    - Learning curve on the VMware tools


    OPTION 3 - WORKING DIRECTLY ON THE SERVER (via remote desktop)

    When to consider: If you have a small company or only 2 people that need to be on the server at one time developing and server administering, you cannot afford the time and software costs of using server virutalization, you do not want developers to have their own server.
    • Note: With Win 2k3 you can only have 2 people working on the server at once (really 3 if you sit someone at the console). You would have to purchase Terminal Services licenses to have more folks login. After 2 people in the box, you will see performance fall of a cliff anyway ...
    Pros:
    - Developers work directly on the server ... less doubt about their code working when deployed
    - Developers do not have to install and maintain Windows 2k3, SQL Server on their PC
    - PCs do not have to be supermachines
    - Quick setup ... only one installation for development
    Cons:
    - Can put other developers out of commission with IIS restarts, errors, or debugging
    - Performance of the development server will decrease with each developer
    - Development tools are installed on server
    - Must almost always have an idea what other developers are doing

    November 5, 2007

    A Collection of Questions and SOLID Answers:

    Q: What is the easiest way to get started with SharePoint 2007?
    A: There is no substitute for hands on experience. Microsoft now offers VHDs (Virtual Hard Disk) that work with Virtual PC or Virtual Server. They are nothing more than a preinstalled version of Windows Server 2003 R2 (for example).

    If you are an administrator I recommend using the Windows 2k3 R2 VHD and installing SharePoint from scratch.

    If you are a developer I recommend the SharePoint 2007 VHD.

    Q: What accounts do I need to setup/create for WSS 3.0 and MOSS 2007?
    A: 1) MSDN, 2) CJ's Blog, 3) harbar

    Q: How should I setup my development environment?
    A: MVN solution

    Q: How do I migrate or backup and restore from one server to another?
    A: 1) MSDN, MSDN (STSADM), More STSADM

    Q: How do I change Database Servers?
    A: 1) STSADM: stsadm -o renameserver -oldservername -newservername
    • Do not expect this to work flawlessly. I recommend extensive testing through CA, backups, deployment, and the site collection.
    2) GUI

    Q: How do I install SharePoint 2007?
    A: Stand Alone OR Complete

    Q: How do I install and configure Reporting Services?
    A: 1) MDSN, 2) Mike's Blog

    Q: How do I setup content deployment from one server to another?
    A: 1) Content Deployment Topology, 2) Content Deployment Explained, 3) Jackie Bodine's Blog - Step by Step instructions

    Q: Why do I receive a "Failed to compare two elements in the array" error message when running a content deployment?
    A: It's all about the features ... 1) Steven Van de Craen's Blog, 2) Soren Nielsen's Blog

    Q: How do I turn off MySites?
    A: Aaron Sailkovski's Blog

    Q: Why are my RSS feeds not showing up?
    A: Tales From The Field

    Q: How do I create and deploy a custom web part?
    A: Lee Dale's Blog

    Q: How do I hide elements of the Document Library toolbar?
    A: 1) Dipper Park's Blog

    Q: How do I uninstall a feature?
    A: stsadm -o uninstallfeature id=[GUID or ID of feature]

    Q: How do I create and deploy a custom user control?
    A: Eric Stallworth

    Q: How do I create an application inside the SharePoint site?
    A: MSDN

    Q: How do I create a customize styles and themes?
    A: (Short answer) I would highly recommend using some type of skinner like this one. Heather Solomon also has a fantastic reference chart.

    Q: How do I customize the Application.Master page?
    A: Sridhar's Blog, MSDN

    Q: How do I customize the Quick Launch menu?
    A: MS Team Blog

    November 1, 2007

    For the Record ...

    My experience with SharePoint started with 2007 and WSS 3.0 in early 2007. Before that I was using MS Content Management Server for roughly 5 years.

    I have my MCTS on the following:

    Microsoft Office SharePoint Server 2007, Configuring (70-630)
    Microsoft Office SharePoint Server 2007, Application Development(70-542)

    I have read several SharePoint books from cover to cover but I find that 80% of my customization and out-of-the-box knowledge has come from blogs and forums.

    This blog will highlight the some of the best posts out there as well as my own work with SharePoint 2007 and WSS 3.0.

    I make no guarantees about any of the information provided on this blog.

    Cheers!